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1  Abstract 

Visual  figures  are  detectable  based  on  a  range  of  spatiotemporal  characteristics  that  differ 
from  surrounding  background.  A  figure  that  corresponds  to  an  ordinary  moving  object  generates 
coherent  two-point  space-time  correlations  related  to  the  first  moment  of  the  luminance 
distribution.  Such  signals  are  readily  detectable  by  the  standard  implementation  of  the 
Hassenstein-Reichardt  elementary  motion  detector  (EMD),  and  as  such  are  referred  to  here  as 
EM.  By  contrast,  figures  defined  by  the  second  or  higher-order  moments,  the  envelope,  or 
variance  of  the  luminance  distribution  evade  detection  by  an  EMD,  yet  are  readily  perceptible  to 
humans  and  non-human  primates,  and  are  thought  to  be  processed  through  distinct  cortical 
streams.  The  project  participants  have  demonstrated  that  flies  process  higher-order  signals  for 
feature  detection,  and  now  others  in  the  field  are  beginning  to  ‘see  the  light’  as  it  were, 
demonstrating  perceptual  capabilities  in  flies  that  rival  any  revealed  in  the  human  psychophysics 
literature.  It  would  appear  that  even  the  most  complex  visual  motion  signals,  including  those  that 
defeat  the  standard  implementation  of  the  EMD,  are  encoded  and  perceived  by  flies  without  the 
computational  overhead  (or  experimental  intractability)  of  cortical  processing.  Under  support  of 
this  grant,  our  group  has  shed  light  on  the  computational  input-output  filters  and  underlying 
neural  foci  of  higher-order  motion  vision  in  flies,  and  begun  to  propose  mechanisms  and  models 
that  extend  current  understanding  of  the  associated  visual  processing.  These  findings  we  expect 
to  be  broadly  applicable  to  all  high-performance  vision  systems,  both  biological  and  engineered. 
In  the  process,  seventeen  manuscripts  have  been  published,  are  under  review,  or  in  production 
for  the  peer-reviewed  literature,  as  a  result  of  the  grant. 

2  Objectives 

The  central  objective  of  the  project  is  an  understanding  of  the  quantitative  systems 
algorithms  and  underlying  neural  circuits  that  enable  flies  to  robustly  code  and  actively  track 
complex  visual  figures  against  cluttered  and  dynamic  visual  backgrounds  in  flight.  In  pursuit  of 
this  objective,  we  aim  to  answer  three  broad  questions: 

1 .  What  are  the  higher-order  spatial  and  temporal  statistical  characteristics  of  visual  features 
that  drive  active  figure  tracking? 

2.  What  is  the  neural  substrate  for  the  detection  of  higher-order  figures,  and  what  are  the 
computational  algorithms? 

3.  What  is  the  relationship  between  figure  detection  and  other  forms  of  visual  motion 
processing,  in  particular  wide-field  motion  processing  and  small  target  detection,  and  in  what 
ways  may  they  be  distinguished? 

There  were  no  changes  to  these  objectives  or  aims  during  the  course  of  the  project. 

3  Accomplishments  /  New  Findings 

3.1  Introduction  and  Background 

The  detection  of  first  order  visual  motion  in  biology  is  well  studied,  both  behaviorally  and 
physiologically.  Even  if  motion  could  theoretically  be  computed  in  a  variety  of  ways,  to  date 
essentially  all  animals  investigated  appear  to  detect  local  motion  in  fundamentally  the  same  way 
(for  a  recent  review,  see  [9]).  In  local  elementary  motion  detectors  (EMDs),  luminance  changes 
from  two  neighboring  inputs  are  correlated  after  delaying  the  signal  from  one  of  the  inputs.  By 
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subtracting  the  output  from  a  mirror  symmetric  subunit,  direction-selective  local  motion 
sensitivity  is  generated.  In  the  vertebrate  visual  cortex,  and  the  fly  lobula  plate,  neurons  with 
large  dendritic  trees  are  believed  to  sample  the  input  from  many  of  such  local  HMDs,  thereby 
generating  wide-field  direction-selectivity  to  the  types  of  optic  flow  generated  during  egomotion. 

The  detection  of  first-order,  or  Fourier,  motion  thus  relies  on  the  presence  of  a  coherent 
correlation  of  luminance  across  space  and  time,  and  forms  the  theoretical  basis  for  the  classic 
EMD  model  of  Hassenstein  and  Reichardt  [24],  However,  general  motion  signals  in  nature  do 
not  always  form  clean  space-time  correlations,  but  may  also  be  comprised  of  higher-order  signals 
related  to  changes  in  contrast  or  texture,  for  example  when  viewing  a  distant,  near-stationary 
scene  through  a  gap  in  the  foliage  while  moving  past  that  gap,  or  viewing  an  object  that  moves  in 
and  out  of  shadows.  The  term  higher-order  motion  refers  to  movement  of  visual  objects  in  which 
internal  first-order  motion  cues  do  not  correspond  with  object  motion.  Such  an  object  might  have 
no  net  motion  energy  (i.e.,  without  Fourier  motion  components  [14]),  or  contain  paradoxical 
motion  cues.  Stimuli  of  the  first  type  are  referred  to  as  drift-balanced,  and  in  our  experiments 
comprise  a  moving  object  containing  an  internal  pattern  that  varies  from  and  replaces  the 
background  as  the  object  moves,  but  which  itself  is  non-moving.  Drift-balanced  stimuli  do  not 
contain  cues  that  are  encodable  by  standard  EMDs  (although  some  degree  of  sensitivity  can  be 
imparted  by  adding  a  front-end  non-linearity  to  the  EMD,  such  as  a  half-wave,  or  full-wave, 
rectifier,  or  a  high-pass  filter  [14][61]. 

Theta  motion  [62]  is  a  second  class  of  higher-order  motion  in  which  a  moving  object 
contains  an  internal  pattern  that  moves  coherently  in  the  direction  opposite  to  the  object  itself  In 
this  case,  an  EMD  would  encode  the  motion  of  the  internal  pattern  (elementary  motion,  EM)  and 
not  the  motion  of  the  object  (figure  motion,  FM).  Theobald  et  al.  [55]  showed  that  fruit  flies 
track  not  only  Fourier  bars,  as  expected  by  a  visual  system  depending  on  EMD-type  input  [10], 
but  that  they  also  track  drift-balanced  bars  and  theta  bars,  showing  that  cortical  processing  is  not 
needed  for  higher-order  motion  sensitivity.  The  demonstrated  sensitivity  to  higher-order  motion 
[55][57]  is  highly  counter-intuitive  as  such  signals  cannot  be  encoded  by  the  standard  EMD  [14], 
and  they  would  thus  not  be  expected  to  be  detectable  at  all  by  motion  processing  computations  in 
the  fly.  The  mere  fact  that  such  computations  are  made  even  by  such  a  small  brain  highlights  the 
importance  of  the  underlying  algorithms  for  high-performance  vision.  Our  research  aims  to 
increase  the  understanding  of  the  behavior  and  neural  circuits  associated  with  these  algorithms. 

Aptekar  and  Frye  have  published  a  critique  of  the  current  literature  that  demonstrates  broad 
parallels  in  the  computations  being  made  in  fly  and  human  vision  [1].  In  particular,  we 
highlighted  motion  stimuli  that  cannot  be  distinguished  by  the  standard  implementation  of  the 
elementary  motion  detector  (EMD),  including  reverse-phi,  plaids,  and  theta  bars,  all  three  of 
which  have  been  first  been  demonstrated  in  humans,  and  now  also  in  flies.  It  would  appear  that 
highly  complex  motion  computations  are  conserved  across  species  separated  by  hundreds  of 
millions  of  years  of  evolution,  implying  that  high  performance  motion  vision  computations  are 
rapid,  compact,  and  prevalent. 

3.2  Experimental  and  Analytical  Techniques 

A  major  area  of  progress  in  this  area  has  been  in  the  application  white  noise  or  m-sequence 
techniques  for  system  identification  to  whole  animal  behavior  in  response  to  the  motion  of 
various  classes  of  visual  figures.  The  Frye  lab  initiated  the  use  of  such  white-noise  techniques  in 
behavioral  studies  of  flies  [13] [56],  and  we  first  applied  them  to  the  problem  of  visual  figure 
detection  under  a  prior  AFOSR  grant  (FA9550-09-1-01 16)  [55].  Under  the  current  grant,  we 
have  significantly  expanded  this  application  to  the  domain  of  figure  detection  and  its  role  in  fly 
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optomotor  responses  [4] [20] [21],  in  the  process  introducing  the  space-time  representations  we 
call  Spatio-Temporal  Action  Fields,  or  STAFs,  which  are  named  for  their  similarity  to  Spatio- 
Temporal  Receptive  Fields  (STRFs)  widely  used  in  electrophysiology.  STAFs  characterize  open- 
loop  motor  response  of  the  fly  to  EM  and  FM  components  of  figure  motion;  the  STAF  concept 
has  now  been  extended  to  characterize  both  compound  figure  and  background  motion,  and  head 
movement  during  figure  tracking  as  well. 

During  pursuit  of  this  work  we  have  developed  numerical  techniques  to  increase  the  accuracy 
of  estimation  of  STAFs,  adapted  and  refined  experimental  /  analytical  techniques  for  use  with 
these  novel  aspects  of  optomotor  figure  responses,  and  (motivated  in  part  by  the  review  process 
on  submitted  manuscripts)  advanced  the  general  understanding  of  their  proper  application  as  well 
as  limitations.  Our  perception  is  that  this  work  has  generated  interest  in  other  vision  science 
laboratories  -  but  in  addition,  a  healthy  skepticism  about  the  overall  approach.  In  response,  we 
have  undertaken  an  effort  to  rigorously  document  the  technique  and  its  theoretical 
underpinnings,  relying  for  examples  on  its  application  our  peer  reviewed  studies.  As  part  of  this 
work,  a  student  in  the  Frye  lab  (P.  Lu)  performed  experiments  with  flies  in  which  optomotor 
responses  were  characterized  for  both  control  animals  and  flies  with  eyes  painted  over,  as  further 
illustration  of  the  technique  and  the  information  it  can  provide.  This  methodology  paper  has  been 
published  (with  associated  Matlab  software)  in  the  journal  Frontiers  in  Neural  Circuits  [2]. 

New  genetic  techniques  available  in  Drosophila  provide  a  powerful  tool  for  targeting  various 
classes  of  neurons  for  the  expression  of  fluorescent  proteins  for  imaging  purposes,  or  of 
inhibitory  ion  channels  for  purposes  of  ‘knocking’  out  cells  by  shunting  of  excitatory  potentials. 
During  the  latter  part  of  the  grant,  we  have  used  two-photon  excitation  calcium  imaging  in 
conjunction  with  such  genetic  techniques  to  target  and  examine  the  responses  of  specific  cell 
types  in  vivo  to  various  types  of  figure  and  edge-related  stimuli. 

Despite  the  genetic  tools  available  in  Drosophila,  larger  flies  are  more  amenable  to 
intracellular  electrophysiology,  which  allows  the  assessment  of  subtle,  but  important,  inhibitory 
and  excitatory  signals.  As  such,  many  questions  are  still  better  addressed  in  the  larger  model 
organisms,  and  thus  we  have  pursued  electrophysiological  study  in  the  element  of  the  grant 
addressed  by  the  Nordstrom  lab.  This  work  has  focused  on  recordings  in  the  lobula  plate  to 
investigate  the  neural  substrate  underlying  higher-order  motion  sensitivity,  and  the  responses  to 
figure-ground  stimuli.  Furthermore,  we  have  searched  for  and  identified  novel,  previously  un¬ 
described  neurons  in  the  fly  lobula  plate. 

3.3  Behavior 

In  psychophysical  studies  through  the  decades,  photoreceptor-sized  displacements  of  a 
luminous  edge  have  revealed  elementary  mechanisms  of  motion  vision  in  insects  and  vertebrates. 
By  contrast,  our  behavioral  work  has  explored  the  extreme  performance  limits  of  motion  vision 
in  flies.  Using  various  ‘virtual  reality’  simulators,  the  Frye  lab  discovered  that  like  humans,  flies 
track  complex  visual  features  that  are  invisible  to  the  canonical  elementary  motion  detector.  We 
devised  a  novel  white  noise  systems  identification  approach  to  demonstrate  parallel  information 
streams  that  encode  diverse  properties  of  visual  features.  We  have  also  showed  that  flies 
suppress  reactions  to  the  visual  surroundings  to  enhance  tracking  of  a  salient  feature.  This 
behavioral  work  has  armed  us  with  algorithms  with  which  to  make  quantitative  predictions  for 
functional  imaging  and  genetic  perturbation  of  the  visual  system  to  probe  computational 
mechanisms.  Other  labs  have  followed  suit,  expanding  the  research  scope  into  higher-order 
motion  vision  in  flies,  which  has  conceptual  parallels  with  psychophysical  work  in  humans. 
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3.3. 1  The  STAF  as  a  characterization  of  behavioral  responses  to  figures 

In  our  behavioral  research,  we  supplemented  the  data  obtained  during  our  initial 
collaboration  under  FA9550-09-1-0116,  and  produced  EM  and  FM  STAFs  for  an  expanded 
sample  size.  As  previously  described,  these  are  based  on  the  reaction  of  tethered  flies  (quantified 
by  the  difference  in  left  and  right  wingbeat  amplitude,  AWBA,  which  is  proportional  to  turning 
torque)  to  horizontally-moving  vertical  bars  displayed  against  static  backgrounds  in  a  flight 
arena.  The  EM  and  FM  STAFs  (illustrated  in  Figure  1  below)  characterize  the  components  of  the 
fly’s  reaction  induced  respectively  by  the  internal  EM  (if  present)  and  by  the  figure  itself,  under 
an  assumption  of  quasilinearity,  including  superposition  of  the  two  components  in  the  total 
output.  Quasilinearity  is  assumed  to  hold  locally,  but  nonlinear  variations  in  the  STAFs  can 
occur  as  figure  position  varies  over  significant  ranges  of  azimuth.  The  analysis  technique  enables 
capture  of  these  spatial  variations  with  sets  of  local  measurements. 


-180  0  180 
Figure  position  (deg) 


3 

03 


e) 


Figure  position  (deg) 


Figure  1:  Wing  steering  Spatio-Temporal  Action  Fields  (STAFs)  for  elementary  motion  (left) 
and  figure  motion  (right),  obtained  by  the  techniques  described  in  the  text.  These 
respectively  describe  local  /  incremental  velocity  impulse  responses  with  respect  elementary 
motion  within  a  figure,  and  motion  of  the  figure  itself,  as  a  function  of  azimuthal  position  of 

the  figure.  The  graphs  to  the  side  and  below  each  false-color  representation  are  cross- 

sections  of  the  STAFs  along  the  time  and  azimuth  axes,  respectively. 

In  additional  work,  we  showed  that  the  generation  of  saccades,  short  applications  of  torque 

by  an  animal  that  in  free  flight  cause  rapid  changes  in  direction,  correlate  closely  with  FM 
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STAFs  (Figure  2).  This  strongly  suggests  that  the  subsystem  that  generates  saccades  under  the 
experimental  conditions  is  driven  by  the  FM  system. 
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Figure  2:  Saccades  are  spatially  mapped  to  the  FM-STAF  and  enhance  active  visual  figure 
tracking.  (A)  Saccades  evoked  toward  figure  movement  in  the  visual  periphery,  but  not  near 
visual  midline.  Two  raw  AWBA  traces  are  shown  for  trials  in  which  a  figure  appears  either 
in  the  center  of  the  arena  (black)  or  in  the  periphery  (gray).  Saccades  are  programmatically 
identified  (orange  dots).  (B)  Distribution  of  saccade  amplitude  under  active  feedback  figure 
control.  Probability  distribution  of  saccade  amplitude  (uncalibrated  voltage  output  of  the 
wing  beat  analyzer,  such  that  negative  values  are  leftward  saccades)  as  a  function  of 
azimuthal  position  of  the  figure.  (C)  Average  Saccade  Interval  as  a  function  of  azimuthal 
position  of  the  figure.  Overlaid  (red)  is  the  spatial  profile  of  the  FM-STAF  from  Figure  2. 
(D)  Average  saccade  amplitude  plotted  as  a  function  of  figure  position  (negative  values  are 
leftward  saccades).  Overlaid  is  the  integral  of  the  ‘center-surround’  function  (red)  fit  to  the 
figure  STAF  spatial  profile,  with  an  integration  constant  chosen  to  cause  a  zero  crossing  at  0 
degrees.  This  function  represents  the  amount  of  accumulated  wing-beat  difference  to  a 
figure  as  it  is  adiabatically  displaced  around  the  azimuth.  For  all  panels,  N=18  flies,  roughly 
20,000  saccades. 


Finally,  in  order  to  validate  the  STAF  methodology,  we  generated  predietions  of  the  torque 
generated  by  the  animals  in  response  to  stimuli  other  than  those  used  to  derive  the  STAFs,  by 
eonvolving  the  eompound  EM  +  FM  STAFs  with  the  eommanded  motion.  These  novel  stimuli 
ineluded  a  Fourier  bar,  a  theta  bar,  and  a  eoherent  motion  bar  with  uneoupled  internal  and  figure 
displaeements  (i.e.,  the  same  elass  of  stimulus  used  for  STAF  derivation,  but  driven  by  a  novel 
pair  of  m-sequenees).  We  also  eomputed  predietions  based  on  the  EM  and  FM  STAFs  alone,  and 
eompared  all  to  measured  data  for  the  same  motion  when  displayed  to  a  sample  of  tethered  flies. 
The  results  are  summarized  in  Figure  3: 
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Figure  3:  (See  prior  page.)  Convolution  of  complex  figure  stimuli  with  both  STAFs  together 
is  sufficient  and  necessary  to  accurately  predict  fly  behavioral  responses.  For  all  panels,  N=5 
flies,  each  having  completed  96  trials.  Simulations  are  averages  of  96  trials.  (A)  An 
illustration  of  the  incremental  construction  of  the  open  loop  simulation  trajectory  to  a 
Fourier  bar.  Space-time  plot  (i)  of  the  stimulus.  EM  system  inputs  (ii)  and  impulse 
responses  (iii).  FM  system  inputs  (iv)  and  step  responses  (v).  Composite  responses 
attributed  to  each  subsystem  (vi)  (FM,  blue;  EM,  red)  plotted  after  each  frame  (transparent) 
and  overall.  The  arrow  labeled  **  indicates  the  time  at  which  the  FM  and  EM  responses  are 
approximately  equal  in  amplitude.  (6)  Space-time  graph  indicating  time  variation  of  a  single 
row  of  arena  pixels.  For  the  Fourier  figure,  internal  elementary  motion  and  fignre  position 
vary  according  to  the  same  triangle  wave  with  a  relative  gain  of  1.  (C)  Mean  behavioral 
responses  (black  line  indicates  mean  fly  response,  grey  envelope  indicates  one  S.E.M.). 

Stimnli  were  convolved  with  each  STAF,  added  together  (magenta).  (D)  Tracking 
trajectories  predicted  by  convolntion  with  either  STAF  independently.  Overlaid  are  mean 
behavioral  responses  as  plotted  in  B.  (E)  Space-time  graph  indicating  a  Theta  fignre,  for 
which  internal  elementary  motion  and  figure  position  are  inversely  coupled  with  a  relative 
gain  of  -1.  (F)  Superposed  STAF  prediction,  plotted  as  in  B.  (G)  Individual  STAF 
predictions,  plotted  as  in  C.  (H)  Space-time  graph,  plotted  as  in  E  for  a  decoupled  Theta  bar 
modulated  by  independent  m-sequences  (see  Fig.  1).  (I)  STAFs  predict  figure  tracking 
responses  to  white  noise  modulated  motion,  plotted  as  in  B  (black  line  indicates  mean  fly 
response,  grey  envelope  indicates  one  S.E.M.).  Steering  response  predicted  with  filter  kernels 
collected  only  at  visnal  midline  is  indicated  in  green.  As  the  figure  diverges  into  the 
periphery,  the  one-dimensional  kernel  prediction  fails  (green  arrow).  (J)  Individnal  STAF 
predictions,  as  plotted  in  D.  For  all  panels,  R  valne  indicates  Pearson  product-moment 
correlation  coefficient  between  mean  fly  responses  (AWBA)  and  STAF  predictions  for  each 
visual  stimulus  as  indicated. 

The  predictive  power  of  the  compound  STAF  is  clear  in  these  experiments  and  analyses, 
confirming  its  validity  as  a  representation  of  the  optomotor  response  of  tethered  flies  to  general 
figure  motion  (for  the  step  sizes  used  in  the  experiments). 

3. 3. 2  Filters  for  visual  figure-ground  discrimination 

Building  upon  our  previous  findings  of  independent  processing  streams  underlying  figure 
tracking  in  flies,  we  turned  our  attention  to  how  figure  tracking  operates  within  the  context  of 
wide-field  background  signals  generated  by  self-motion.  We  aimed  to  characterize  the 
spatiotemporal  properties  of  optomotor  responses  to  both  wide-field  ground  motion  and  figure 
motion  for  varying  azimuthal  positions  of  the  figure.  Adapting  our  previous  white-noise 
methodology,  we  computing  velocity  impulse  response  filters  to  figure  motion  and  ground 
motion  parameterized  over  the  visual  azimuth,  and  generated  STAFs  for  each.  These  STAFs 
demonstrate  how  the  temporal  dynamics  of  incremental  steering  responses  to  velocity  impulses 
of  a  small  visual  figure  and  simultaneous  velocity  impulses  of  the  wide-field  ground  vary  as  a 
function  of  the  spatial  location  of  figure  motion  on  the  visual  azimuth. 

We  first  demonstrated  that  flies  can  actively  target  visual  figures  under  closed-loop  feedback 
control  even  under  the  limit  in  which  any  incremental  motion  was  matched  by  ground  motion 
oriented  in  the  opposite  direction.  Figure  tracking  is  so  robust  that  is  impervious  to  wide-field 
motion  signals  that  move  completely  out  of  phase  with  the  fly’s  own  steering  commands.  This 
can  only  be  achieved  by  some  mechanism  of  inhibition  or  suppression  of  wide-field  stabilization 
reflexes.  By  using  m-sequences  to  characterize  the  spatial  and  temporal  dynamics  of  figure  and 
ground  motion  (graphically  represented  by  the  STAFs),  we  show  functionally  how  this  is 
accomplished.  In  particular,  we  show  that  for  figures  defined  only  by  their  relative  motion,  (1) 
flies  actively  fixate  a  visual  figure  against  a  moving  ground  even  when  the  motion  signals 
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directly  oppose,  (2)  figure  and  ground  STAFs  collected  simultaneously  indicate  distinct  spatial 
tuning  and  dynamics  (Figure  4),  (3)  the  presence  of  a  figure  is  required  for  active  suppression  of 
the  wide-field  optomotor  stability  response  (Figure  4),  (4)  STAFs  predict  fly  responses  to 
compound  figure-ground  stimuli.  This  work  has  been  published  in  the  Journal  of  Experimental 
Biology  [2], 
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Figure  4:  Wing  steering  Spatio-Temporal  Action  Fields  (STAFs)  collected  from  a  single  fly, 
and  average  STAFs  collected  from  a  group  of  flies.  STAFs  reveal  the  robust  figure  tracking 
response  on  visual  midline  and  the  spatial  inhibition  apparent  in  ground  responses  within 
the  region  of  strongest  figure  sensitivity  (‘notch’  in  Ground  STAFs  below  red  arrowheads). 
We  show  direct  evidence  of  inhibition  of  the  ground  stabilization  system  when  a  figure  is 
occupying  the  forward  field  of  view,  and  is  being  actively  tracked;  in  control  experiments, 
replacing  the  uncorrelated  figure  motion  stimulus  with  a  figure-sized  patch  of  uniform  gray 
does  not  provoke  the  spatial  inhibition  seen  here.  AWBA  refers  to  the  difference  in  wing  beat 
amplitude  across  the  left  and  right  wings  (units  of  uncalibrated  sensor  voltage). 


If  flies  suppress  optomotor  steering  responses  to  wide-field  motion  while  tracking  visual 
figures,  does  that  mean  that  they  are  generally  unable  to  stabilize  their  visual  gaze?  Virtually  all 
reports  that  record  wing  kinematics  and  head  motions  together  demonstrate  that  the  two  are 
tightly  correlated  [16]  [48],  Yet,  none  investigate  how  the  visual  gaze  (head  orientation)  is 
controlled  during  figure-ground  discrimination.  We  used  real-time  tracking  video  to  construct 
linear  filters  representing  head  movements  in  response  to  m-sequence-mediated  velocity 
impulses  of  figure  and  ground  motion,  under  similar  conditions  as  described  above.  The  results 
were  surprising  and  informative.  We  note  first  that  under  identical  conditions,  wing  kinematics 
share  the  overall  control  effort  to  simultaneously  steer  toward  a  salient  visual  figure  and  also 
stabilize  perturbations  to  the  wide-field  ground.  Thus,  we  recover  STAFs  for  the  figure  and 
ground  motions  that  are  distinct  in  space  and  time  (Figure  4).  Head  kinematics  do  not  split  the 
control  effort,  but  rather  seem  to  be  completely  overridden  by  ground  motion.  Thus,  we  recover  a 
STAF  only  for  ground  motion,  but  not  for  concomitant  figure  motion  (Figure  5,  left  and  center). 
In  the  absence  of  uncorrelated  ground  motion,  however,  the  heads  of  flies  robustly  track  figure 
motion  (Figure  5,  right).  These  results  show  for  the  first  time  that  the  control  of  visual  gaze  can 
be  uncoupled  from  the  control  of  steering  torque,  but  only  by  the  relative  motion  of  visual 
figures  on  ground.  These  findings  are  elaborated  in  a  second  paper  in  the  Journal  of 
Experimental  Biology  [3]. 
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Figure  5:  Spatio-temporal  action  fields  (STAFs)  for  head  movements  collected  for  n=27  flies. 

Head  STAF  for  figure  motion  (left)  and  for  ground  motion  (right).  Note  the  lack  of  responses 
to  figure  motion  presented  with  concomitant  ground  motion  (left),  and  the  lack  of  any 
suppression  of  the  ground  response  (center),  even  though  the  heads  follow  figure  motion 
with  there  is  no  ground  stimulus  (right). 

In  summary,  our  findings  make  significant  headway  in  our  understanding  of  simple 
algorithms  for  figure-ground  discrimination,  depending  critically  on  the  spatial  tuning  of  figure 
tracking  we  reported  previously  [4].  Indeed  the  finding  that  figure  tracking  is  broadly  robust  to 
head  movements  [4]  is  now  extended  to  ground  motion  [3], 

3.3.1  A  figure  edge  is  coded  by  the  higher-order  FM  pathway 

Tethered  flies  fixate  a  contrasting  vertical  bar  frontally  under  closed- loop  feedback 
conditions  [22],  For  a  wide  bar,  fixation  is  bistable,  with  two  prominent  peaks  in  the  distribution 
of  the  azimuthal  position  of  the  bar’s  edges,  suggesting  that  the  salient  component  of  the  figure  is 
its  edge  rather  than  center  of  mass  [26][34][58].  We  extended  those  studies  by  using  a  white- 
noise  approach  to  map  out  the  underlying  motion  and  position  dependent  systems  that  govern 
figure  tracking  [4]  and  measured  STAFs  for  figures  of  30  and  90  degree  widths. 

Comparing  the  STAFs  reveals  that  the  two  elements  of  figure-tracking  -  Elementary  Motion 
(EM)  impulse  responses  and  Figure  Motion  (FM)  step  responses  -  are  affected  differently  by 
figure  width  (Figure  6).  Consistent  with  a  model  of  EM  processing  that  relies  on  the  summation 
of  local  motion  detector  inputs  [23] [36] [50],  the  EM  STAF  broadens  with  figure  width  (Figure 
6i),  since  more  local  motion  detectors  would  be  stimulated  by  the  larger  figure.  In  contrast,  the 
step  response  of  the  FM  STAF  splits  for  increasing  figure  width  (Figure  6ii,  arrowheads), 
indicating  that  the  strongest  steering  effort  is  produced  when  the  edge  of  the  90-degree  figure  is 
aligned  on  midline.  This  supports  the  hypothesis  that  flies  preferentially  track  the  edge,  or  optical 
disparity,  produced  by  a  figure.  [46],  This  work  is  included  in  a  manuscript  currently  under 
review  at  the  Journal  of  Neuroscience. 


FA9550-12-1-0034  Final  Report 


Head  Figure  STAF  Head  Ground  STAF  Head  Figure  STAF 

(no  ground  motion) 


11 


FA9550-12-1-0034  Final  Report 


30  MAY  2014 


30“EM-STAF  30TM-STAF 


Position,  azimuth  (deg) 


90TM-STAF 


A  A 


Figure  6:  Spatio temporal  action  fields  (STAFs)  for  figure  motion  (FM)  and  small-field 
elementary  motion  (EM)  for  a  30-degree  wide  (i)  and  90-degree  wide  (ii)  figure.  The  lobe 
separation  for  the  90-deg  FM-STAF  (arrowheads)  corresponds  to  the  functional  separation 
of  the  two  edges  of  the  bar  by  the  FM  system. 


3. 3. 2  Figure  tracking  and  threat  avoidance  by  the  FM  system 

The  ability  to  use  vision  to  perform  a  broad  set  of  ethologically  important  behaviors  is 
critical  for  flying  insects.  Importantly,  the  visual  system  must  be  able  to  robustly  discriminate 
between  visual  figures  that  are  attractive — mates,  perches,  food  resources — and  those  that  are 
aversive — ^predators.  However,  this  provides  a  challenge  to  the  overall  system  architecture  of  the 
optomotor  system:  to  the  extent  that  primitive  visual  navigation  algorithms  can  be  shared  across 
behavioral  contexts,  the  systems  architecture  can  be  streamlined.  However,  to  the  extent  that 
different  contexts  necessitate  dramatically  different  behaviors — fleeing  a  predator  vs.  pursuing  a 
mate — a  single,  common  architecture  will  constrain  behavioral  flexibility. 

In  this  work,  we  employ  the  spatiotemporal  action  field  (STAF)  methodology  to  break  down 
the  fruit  fly’s  response  to  figures  with  different  aspect  ratios.  We  demonstrate  how  the  existence 
of  parallel  processing  streams  for  visual  motion  information  that  contribute  in  combination  to  the 
overall  steering  effort  can  accomplish  a  complex  task,  like  avoidance  of  small  objects,  without 
wholesale  dispensing  with  the  influence  of  low-level  programs — like  EM  tracking.  Further,  we 
reinforce  the  importance  of  saccadic  tracking  to  visually-guided  navigation.  While  saccades  have 
long  been  understood  as  the  primary  navigational  unit  in  free-flight  and  tethered  flight  where 
flies  are  allowed  an  axial  degree  of  freedom,  torque  spikes  are  often  neglected  in  rigidly-tethered 
flight.  This  study  firmly  identifies  the  saccadic  tracking  apparatus  as  the  primary  mechanism  of 
small-object  avoidance. 

Consistent  with  the  model  of  elementary  motion  processing  being  driven  by  a  distributed  set 
of  elementary  motion  detectors  (HMDs)  likely  instantiated  at  the  level  of  the  individual 
ommatidial  facet,  we  demonstrated  that  the  degree  of  EM  response  to  small  objects  decreased 
monotonically  with  the  vertical  extent  of  the  object.  As  the  amount  of  small-field  texture 
decreased,  corresponding  to  a  smaller  overall  EM  motion  energy,  the  EM  response  reduced 
correspondingly. 

Consistent  with  the  observations  of  Maimon  et  al.  [35],  we  found  that  the  FM-response  of  the 
fly  inverts  as  the  object  is  made  shorter  (Figure  7).  This  inversion  takes  two  forms:  first  by  the 
comparative  reduction  in  amplitude  of  the  FM  STAF  in  the  frontal  visual  field  as  compared  to 
the  rear-field,  and  second  by  the  ultimate  inversion  of  the  FM  STAF  in  the  frontal  visual  field 
when  the  object  is  sufficiently  short.  One  of  the  most  conceptually  challenging  parts  of  the 
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Maimon  study  was  the  observation  of  small  objeet  ‘anti-fixation’  where  a  tethered  fly  under 
closed- loop  conditions  was  shown  to  stably  fixate  a  small-object  in  the  rear  field  of  view  with 
some  finite  probability.  This  finding  is  consistent  with  the  gain  of  the  optomotor  response 
relative  to  the  avoided  figure  having  a  positive  sign,  which  would  allow  stabilization  of  the 
figure  at  a  particular  location  in  the  visual  field.  We  note  that  the  relatively  unchanged  and  yet 
still-positive  sign  of  the  STAF  in  the  rear  visual  field  makes  the  rear  field  a  comparatively  more- 
stable  pole  of  fixation  than  the  frontal  field,  which  is  consistent  with  anti-fixation.  The  reduction 
of  the  STAF  in  the  frontal  field  makes  this  frontal  pole  comparatively  less  stable  and,  once  it 
actually  inverts,  the  frontal  pole  becomes  unstable.  This  demonstrates  that  the  position-sensitive 
component  of  the  fly’s  steering  to  a  small  object,  the  FM  STAF,  has  a  distinctly  different 
character  than  its  EM  STAF. 
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Figure  7:  Spatiotemporal  action  fields  (STAFs)  for  moving  bars  as  the  bar  height  decreases. 
Below  the  sets  of  STAFs  are  shown  their  values  averaged  over  time  as  a  function  of  azimuth. 
These  results  illustrate  the  decrease  and  gradual  inversion  of  the  FM  STAF  in  the  frontal 
visual  field,  with  a  corresponding  reduction  in  frontal  fixation  stability  eventually  leading  to 
loss  of  the  stable  fixed  point  at  front-center,  and  the  onset  of  anti-fixation  behavior.  The  EM 
STAF,  conversely,  simply  decreases  in  amplitude  as  the  total  (first-order)  motion  energy 
decreases  in  proportion  to  bar  height. 


3.3.3  STAFs  in  the  frequency  domain  and  figure  tracking  microsaccades 

One  aspect  of  the  STAFs  that  limits  the  intuition  that  can  be  gained  from  them,  as  well  as 
their  application  to  engineering,  is  that  they  are  measured  in  the  time  domain.  It  would  be  useful 
for  both  purposes  to  measure  the  frequency  response  associated  with  ground  and  figure  tracking 
across  the  visual  field  (although  care  must  be  taken  in  applying  frequency-domain 
representations  to  engineering  and  modeling,  due  to  the  inherent  nonlinearities  in  the  system 
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which  will  become  significant  under  large  variations  in  figure  velocities).  A  new  postdoc  in  the 
Frye  lab,  Jean-Michel  Mongeau,  recruited  from  Bob  Full’s  lab  at  UC  Berkeley,  has  measured  the 
power  spectral  density  of  time-domain  STAFs  (Figure  8A,B).  Comparing  the  time  and  frequency 
domain  demonstrates  that  when  a  figure  is  near  the  frontal  field  of  view,  the  steering  response  to 
the  figure  contains  higher  frequency  content,  and  correspondingly  the  frequency  response  to 
ground  motion  filters  lower  frequencies  (Figure  8B).  In  addition  to  high  frequency  smooth 
steering  toward  a  frontal  visual  figure,  previous  work  suggested  that  rapid  saccadic  turns  are 
oriented  elicited  when  the  figure  is  positioned  in  the  visual  periphery,  where  the  azimuthal 
integral  of  the  FM-STAF  is  maximal  [4],  However,  these  experiments,  by  design,  were 
performed  on  rigidly  fixed  flies  that  cannot  rotate  and  thus  cannot  experience  the  full  array  of 
visual  and  mechanosensory  feedback  elicited  during  normal  flight  conditions.  To  explore  the 
hypothesis  the  high  frequency  figure  tracking  and  body  saccades  combine  in  active  figure 
tracking,  we  used  a  different  flight  simulator  apparatus  in  which  the  fly  is  tethered  to  a  steel  pin 
and  suspended  in  a  magnetic  field,  free  to  pivot  in  the  yaw  plane  [6]  [16].  For  a  figure  revolving 
at  constant  velocity,  a  fly  demonstrates  very  clear  ‘microsaccades’  in  body  orientation  (Figure 
8C).  We  have  since  demonstrated  that  flies  robustly  stabilize  their  heading  between  saccades 
(Figure  8D),  and  that  the  microsaccades  are  elicited  near  the  maxima  of  the  FM-STAF  (Figure 
8E,F).  These  microsaccades  are  specialized  for  active  figure  tracking,  and  are  qualitatively  and 
quantitatively  distinct  from  the  larger  slower  body  saccades  elicited  by  the  visual  surroundings 
[6].  We  are  currently  exploring  the  neuronal  mechanisms  that  trigger  body  saccades  upon 
detection  and  discrimination  of  a  visual  figure  from  the  moving  ground. 
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Figure  8:  A)  Time-domain  figure-ground  spatio-temporal  active  field  (STAF).  w  =  14  flies. 

The  time-domain  STAF  reveals  that  when  the  figure  is  near  the  visual  midline,  flies  track  the 
figure,  but  when  the  figure  is  in  the  periphery,  flies  track  the  ground.  B)  Power  spectral 
density  of  time-domain  STAF.  When  analyzing  the  STAF  in  the  frequency  domain,  we 
determined  that  flies’  overall  tracking  effort  is  a  low-pass  filter  in  transforming  the  stimulus. 

For  the  figure  transform,  when  the  figure  is  near  visual  midline,  flies  pass  higher  frequencies 
than  when  the  figure  is  in  the  periphery.  In  contrast,  for  the  ground  transform,  when  the 
figure  is  near  the  visual  midline,  flies  pass  lower  frequencies  when  the  figure  is  near  visual 
midline  but  pass  higher  frequencies  when  the  figure  is  in  the  periphery.  C)  Typical 
trajectory  of  a  magnetically-tethered  fly  (red)  tracking  a  vertical  figure  (black)  moving  on  a 
random  background.  The  response  is  dominated  by  body  saccades.  D)  Inter-saccade 
trajectories  of  flies  during  bouts  of  figure  tracking  around  the  arena.  Between  saccades, 
there  is  little-to-no  smooth  tracking  via  the  optomotor  reflex.  Thick  black  line  is  the  arena 
reference  angle.  Thick  blue  line  is  the  mean  trajectory  of  flies  between  saccades.  Thin  blue 
lines  are  individual  trials.  E)  Saccade-triggering  errors  between  the  fly’s  heading  and  the 
figure  position.  Body  saccades  are  maximally-triggered  near  angles  of  ±90°  {n  =  20  flies).  F) 
Mapping  of  torque  spikes  to  figure  position  for  figure-ground  STAF  experiment  in  rigidly- 
tethered  flies.  These  results  show  that  the  highest  density  of  torque  spikes  is  near  ±90°, 
consistent  with  data  in  magnetically-tethered  flies  {n  =  14  flies). 

In  upcoming  work,  we  intend  to  expand  our  understanding  about  figure  elicited 
microsaccades  with  in  vivo  imaging  physiology.  We  have  begun  a  screen  for  Gal4-lines  that 
might  reasonably  participate  in  the  motor  control  of  microsaccades  in  the  pre-motor  pathways 
outside  of  the  lobula  plate.  In  doing  so,  we  are  equipping  our  two-photon  imaging  system  with  a 
stage  that  allowed  for  imaging  during  tethered  flight.  We  are  also  developing  a  computational 
control-theoretic  model  to  generate  specific  quantitative  hypotheses  about  the  sensory 
information  that  would  be  expected  within  the  control  loop  of  figure  microsaccades. 
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3.4  Electrophysiology 

Our  behavioral  work  has  established  that  figure  tracking  is  subserved  by  two  parallel, 
superposed  motion-processing  streams,  one  coding  for  elementary  motion  (EM)  cues  (i.e., 
motion  that  is  spatiotemporally  luminance-correlated  and  thus  directly  encodable  by  HMDs),  and 
the  second  for  figure  position  via  figure  motion  (FM)  cues  that  may  be  higher-order  in  nature.  In 
the  electrophysiology  element  of  the  project  we  thus  set  out  to  investigate  what  neurons  may 
underlie  the  demonstrated  FM  sensitivity  in  flies.  We  began  with  the  well-known  lobula  plate 
tangential  cells  (LPTCs),  which  synapse  directly  with  descending  neurons  that  control  the 
behavioral  optomotor  response,  and  whose  responses  to  wide-field  elementary  motion  have  been 
widely  studied  (see,  e.g.,  [8] [9]).  Interestingly,  the  LPTCs  that  we  chose  for  initial  investigation 
(belonging  to  the  HS  and  VS  systems)  have  heretofore  been  assumed  to  be  wide-field  integrators 
of  EMD  outputs  -  so  any  response  to  higher-order  motion  in  these  neurons  would  be  of 
particular  interest. 

Recent  work  from  the  Borst  group  [5]  also  supports  this  general  subdivision,  into  EM  and 
FM  processing  streams.  Interestingly,  however,  the  authors  suggest  that  the  lobula  plate  is 
entirely  dedicated  to  the  computation  of  wide-field  EM  and  serves  no  role  in  the  detection  of 
figure  motion.  This  conclusion  is  in  stark  contrast  with  results  from  our  own  work. 

3.4. 1  Sensitivity  to  higher-order  motion  in  the  LPTCs 

To  investigate  the  sensitivity  of  fly  LPTCs  to  higher-order  motion,  we  recorded  intracellular 
responses  in  the  hoverfly  lobula  plate.  We  quantified  responses  to  moving  bars,  including  those 
corresponding  to  ordinary  moving  objects  (Fourier  bars),  but  also  to  second-order  bars  in  which 
the  internal  pattern  dynamics  did  not  correspond  to  the  figure’s  motion.  In  the  latter  cases,  the 
EMDs  that  are  supposed  to  underlie  LPTC  responses  are  not  expected  to  give  net  responses  that 
track  the  figure  velocity.  Figure  9  depicts  the  stimuli  used  and  responses  of  a  simple  EMD-based 
model: 
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Figure  9:  Data  in  column  A  relate  to  Fourier  bars,  column  B  to  drift-balanced  bars,  column 
C  to  theta  bars,  and  column  D  to  small-field  motion.  The  first  row  shows  example  space-time 
plots  for  each  stimulus,  where  the  vertically  oriented  bar  (6x37°)  moved  horizontally  at  2 
pixels  per  frame,  corresponding  to  ca.  50°/s,  for  375ms  before  switching  direction,  except  in 
column  D  where  the  figure  itself  remained  stationary  while  the  internal  pattern  moved.  In 
these  plots,  the  abscissa  represents  time  and  the  ordinate  azimuth  relative  to  front  and 
center,  and  the  displayed  pattern  (which  is  one-dimensional  since  it  consists  of  uniform 
vertical  stripes)  is  represented  in  black  and  white  as  a  function  of  position  and  time.  Data  in 
the  second  row  show  the  summed  output  in  arbitrary  units  of  a  standard  EMD  in  response 
to  100  randomly  generated  stimuli  of  each  class.  There  is  no  net  response  to  drift-balanced 
motion,  whereas  for  theta  motion,  it  is  the  internal  pattern  velocity  rather  than  the  figure 
that  is  followed. 

Defining  ‘preferred  direetion’  as  the  direetion  of  first-order  motion  that  produees  maximum 
depolarization,  we  surprisingly  found  regions  of  high  sensitivity  (i.e.,  depolarization)  in  the 
lobula  plate  eell  HSN  when  any  type  of  bar  moved  in  the  preferred  direetion  (Figure  10).  For  the 
theta  bar,  this  means  that  the  net  neural  response  is  opposite  to  the  expeeted  elementary  motion 
response.  (This  eharaeteristie,  however,  does  not  prevail  for  anti-preferred  direetion  motion.)  The 
FM  sensitivity  was  partieularly  robust  along  the  visual  midline  of  the  fly  (shaded  area.  Figure 
10),  supporting  a  possible  role  in  the  behavior  when  bars  eontaining  higher-order  motion  eues 
are  aetively  fixated  in  the  frontal  visual  field. 
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Figure  10:  A)  When  a  Fourier  bar  moves  in  the  preferred  direction  (red)  over  a  random 
background  with  the  same  contrast  and  spatial  characteristics  as  the  bar  itself,  the  neuron 
responds  with  a  depolarization  that,  when  plotted  as  a  function  of  azimuth,  reflects  the  shape 
of  the  underlying  male  HSN  receptive  field  (inset).  The  response  to  anti-preferred  direction 
motion  (blue)  is  inverted  and  weaker  in  amplitude,  but  follows  the  same  general  shape.  B)  In 
a  drift-balanced  bar  there  is  no  first-order  motion  in  the  internal  pattern  (EM=0)  as  the 
figure  sweeps  across  the  visual  field.  The  two  data  traces  show  the  responses  to  preferred 
direction  FM  (red)  and  anti-preferred  direction  FM  (blue).  C)  The  two  data  traces  show  the 
responses  to  theta  motion  where  FM  is  in  the  preferred  direction  (red)  and  anti-preferred 
direction  (blue).  D)  A  quantitative  comparison  of  the  mean  responses  to  the  three  bars,  10° 
ipsilateral  of  the  visual  midline  (shaded  in  A-C).  Red  bars  show  responses  to  preferred 
direction  FM,  blue  responses  to  anti-preferred  direction. 

Interestingly,  we  found  similar  behavior  in  a  number  other  HS  and  VS  neurons:  when  any 
figure  moves  in  a  neuron’s  preferred  direetion,  the  neurons  tend  to  depolarize  even  for  drift- 
balaneed  and  theta  bars. 

We  quantified  the  sensitivity  to  pure  EM  (small-field  motion)  and  to  pure  FM  (a  drift- 
balaneed  bar)  aeross  the  visual  field  (spatial  sensitivity)  and  time  (temporal  profile).  This  showed 
the  FM  sensitivity  to  be  a  lot  more  ‘jagged’  than  the  EM  sensitivity  (Figure  11).  We  found  no 
large  differenees  in  the  response  onset  to  motion  direetion  reversals,  but  in  response  to  FM  the 
membrane  polarization  builds  up  a  lot  more  slowly  than  it  does  for  EM.  This  again  suggests  a 
different  underlying  input  behind  the  observed  FM  sensitivity  in  LPTCs. 
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Figure  11:  The  responses  to  EM  and  FM  from  different  parts  of  the  visual  field,  displayed  as 
a  function  of  time  since  direction  reversal  on  the  y-axis,  and  the  corresponding  azimuthal 
location  on  the  x-axis.  The  top  rows  show  example  space-time  plots.  Both  stimuli  were 
vertical  bars  (8x75°).  The  FM  bar  moved  horizontally  at  2  pixels  per  frame,  corresponding 
to  ca.  50°/s,  for  375ms  before  switching  direction.  The  EM  bar  remained  stationary  while  the 
internal  pattern  moved.  Left  column:  As  an  FM  bar  moves  to  the  right,  i.e.  in  the  anti¬ 
preferred  direction  of  the  recorded  neuron,  it  leaves  the  ipsilateral  visual  field  and  moves 
into  the  contralateral  visual  field  during  its  trajectory.  As  the  bar  crosses  the  midline  (white 
dashed  line)  the  membrane  potential  returns  to  pre-stimulus  levels.  When  the  bar  changes 
direction  and  moves  in  the  preferred  direction  of  the  neuron,  HSN  depolarizes  as  the  bar 
enters  the  ipsilateral  visual  field.  The  blue  and  red  dashed  lines  indicate  the  part  of  the  visual 
field  covered  by  bar  motion  at  each  point  in  time.  To  the  right  of  each  spatio-temporal 
response  plot  we  show  the  response  averaged  across  space,  as  a  function  of  time.  The  red  and 
blue  arrows  indicate  the  time  at  which  the  response  changes  sign  after  direction  reversal. 

The  time-to-peak  response  is  longer  for  FM  than  for  EM:  90ms  and  190ms  for  FM 
(arrowheads),  and  33ms  and  30ms  for  EM. 

These  findings  were  published  in  the  Proeeedings  of  the  National  Aeademy  of  Seienees  [32], 
with  key  points  are  highlighted  above. 

3. 4. 2  Figure-ground  discrimination  in  hoverfly  LPTCs 

In  this  projeet  we  have  shown  that  figure-ground  behavior  ean  be  broken  down  into  separate 
eomponents  that  eode  for  the  motion  of  the  baekground,  the  elementary  motion  assoeiated  with 
the  bar  and  the  position  of  the  bar,  respeetively,  as  deseribed  in  detail  above.  Interestingly, 
previous  work  has  shown  that  loeal  features  have  a  strong  influenee  on  the  response  properties  of 
HS  eells.  For  example,  during  optie  flow  reeonstrueted  from  aetual  flight  paths,  the  translation  of 
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near-by  features  generates  strong  transient  responses  in  blowfly  HS  neurons  [7][31][33].  In 
hoverfly  HS  neurons,  high-contrast,  vertically  oriented  features  generate  strong  contrast  gain 
reduction  that  reduces  the  response  to  subsequent  low-contrast  stimuli  [41],  Furthermore,  pattern 
noise,  i.e.  the  response  to  individual  features  within  a  scene,  is  stronger  in  hoverfly  HSN,  with  its 
small  receptive  field,  than  in  HSNE  [42],  which  has  a  more  LPTC-typical,  laterally  extended 
receptive  field  [38],  Particularly  the  HSN  response  to  vertically  elongated  features  is  enhanced, 
suggesting  that  it  may  be  involved  in  forward  fixation  behaviors  [42], 

To  further  investigate  responses  to  figure  motion  in  the  LPTCs,  we  recorded  intracellular  HS 
responses  to  salient  bars  moving  with  or  against  a  broadband  background  moving  at  the  same 
velocity,  and  compared  the  responses  to  bar  and  background  motion  on  their  own.  The  bars  were 
narrower  than  the  acceptance  angle  of  a  single  ommatidium,  ensuring  that  the  background 
motion  constitutes  the  spatially  dominating  component  of  the  visual  stimulus.  When  moving  in 
the  preferred  direction  over  a  mean  luminance  background,  the  bar  generates  a  strong 
depolarization  of  male  HSN  (black.  Figure  12 A),  and  a  hyperpolarization  in  the  anti-preferred 
direction  (black.  Figure  12B). 

The  broad-band  background  image  (for  description,  see  [39])  also  drives  the  neuron  strongly 
in  both  the  preferred  and  anti-preferred  direction  (grey  data.  Figure  12).  Despite  the  bar’s  narrow 
spatial  extent,  it  generates  a  stronger  response  than  the  background  when  it  moves  through  the 
HSN  receptive  field  (compare  black  and  grey  responses.  Figure  12),  suggesting  strong  neural 
amplification  of  the  response  to  the  bar. 

We  examined  bar-background  responses  by  displaying  the  bar  over  the  background.  When 
the  background  moves  in  the  preferred  direction,  and  the  bar  passes  through  the  receptive  field  in 
the  preferred  direction  (red.  Figure  12A),  the  response  shape  resembles  the  bar  response  (black. 
Figure  12 A)  more  than  the  background  response  (grey.  Figure  12 A).  When  the  background 
moves  in  the  opposite  direction  to  the  bar,  the  response  also  largely  resembles  the  bar  response 
(compare  black  and  blue.  Figure  12 A).  Indeed,  when  the  bar  moves  through  the  receptive  field, 
HSN  is  strongly  depolarized  (blue.  Figure  12  A),  despite  99%  of  the  visual  stimulus  moving  in  its 
anti-preferred  direction.  The  strong  correlation  with  the  bar  response  is  particularly  evident  in  the 
400ms  surrounding  the  receptive  field  hot-spot  (Figure  12C). 

When  the  bar  moves  in  the  neuron’s  anti-preferred  direction,  the  response  to  bar-background 
motion  also  resembles  the  response  to  bar  motion  (black.  Figure  12B)  more  than  the  response  to 
background  motion  (grey.  Figure  12B),  if  the  background  moves  in  the  anti-preferred  direction 
(cyan.  Figure  12B).  However,  when  both  the  background  moves  in  the  preferred  direction 
(green.  Figure  12B)  the  response  does  not  reach  the  same  level  of  hyperpolarization  as  during  bar 
motion  alone  (black.  Figure  12B). 
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Figure  12:  HSN  background  responses  to  figure-ground  motion.  A.  The  response  of 
male  HSN  to  a  black  bar  moving  in  the  preferred  direction  over  a  mean-luminance 
background  (black,  N=5).  The  grey  data  show  the  responses  to  the  background 
pattern  moving  in  the  preferred  or  anti-preferred  direction.  The  red  data  show  the 
response  when  the  bar  and  the  background  both  move  in  the  preferred  direction, 
with  no  velocity  difference.  The  blue  data  show  the  response  to  the  bar  moving  in 
the  preferred  direction  over  the  background  moving  in  the  anti-preferred  direction. 

B.  The  male  HSN  response  to  bar  motion  in  the  anti-preferred  direction  (black). 

The  cyan  data  show  the  response  to  both  bar  and  background  moving  in  the  anti¬ 
preferred  direction  and  the  green  data  the  response  to  the  bar  moving  in  the  anti¬ 
preferred  direction  over  the  background  moving  in  the  preferred  direction.  The 
bars  under  the  trace  show  the  peri-stimulus  durations  for  bar  and  background 
motion.  The  data  were  low-pass  filtered  for  display.  C.  A  magnification  of  the  data 
in  panel  A  for  the  400ms  used  for  quantitative  analysis  in  subsequent  figures.  The 
frames  of  the  pictograms  are  color  coded  to  illustrate  the  four  combinations  (red: 
barpi-gj— backgroundpref,  blue,  barpi-gj-backgroundantj.pi-ef,  green.  bargi,^i_pi<gj— backgi<oiii,(ipi<gf, 
cyan  bara„a.prerbackgrounda„a.pref). 

To  quantify  these  results,  we  display  the  response  to  bar-background  motion  as  a  function  of 
the  response  to  bar  motion  alone,  during  the  400ms  the  bar  passed  the  receptive  field  center  (as 
magnified  in  Figure  12C).  This  analysis  shows  a  strong  correlation  between  the  bar-background 
response  and  the  bar  response  (Figure  13 A,  D;  the  data  are  color  coded  to  illustrate  the  four 
conditions),  and  only  a  poor  correlation  with  the  background  response  (Figure  13B,  D).  To  test 
whether  the  response  to  bar-background  motion  represents  a  summation  of  the  individual 
responses  to  bar  motion  and  background  motion,  respectively,  we  display  the  response  to  bar- 
background  motion  as  a  function  of  the  sums  of  the  responses  to  bar  motion  alone  and 
background  motion  alone  (Figure  13C,  D).  This  analysis  suggests  that  the  linear  sum  of  the 
responses  to  bar  motion  and  background  motion,  respectively,  only  gives  a  good  correlation  with 
the  bar-background  response  when  the  background  moves  in  the  preferred  direction  and  the  bar 


22 


FA9550- 12- 1-0034  Final  Report 


30  MAY  2014 


in  the  anti-preferred  direction  (green,  Figure  13C,  D).  In  all  other  cases,  the  response  to  bar- 
background  motion  is  most  similar  to  the  response  to  bar  motion  alone  (black  data.  Figure  13D). 
This  suggests  that  the  bar  response  dominates  in  compound  responses  to  bar-background  motion. 
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Figure  13:  HSN  background  responses  are  suppressed  by  motion  of  a  salient  bar.  A. 

The  response  to  figure-ground  motion  as  a  function  of  the  response  to  bar  motion 
alone,  for  the  400ms  surrounding  the  receptive  field  center.  The  data  points  are 
color  coded  according  to  the  pictograms  and  data  in  Figure  12B.  The  response  to 
figure-ground  motion  as  a  function  of  the  response  to  background  motion.  C.  The 
response  to  figure-ground  motion  as  a  function  of  the  summed  responses  to  bar 
motion  and  to  background  motion  on  their  own.  D.  The  difference  of  the  mean 
response  (in  mV)  between  the  response  to  figure-ground  motion  and  bar  motion 
alone  (black  fills),  background  motion  alone  (BG,  grey  fills),  and  the  sum  of 
background  and  bar  motion  responses  (white  fills).  The  frames  of  the  bars  are  color 
coded  to  illustrate  the  four  combinations  of  motion  directions. 

Among  LPTCs,  the  hoverfly  HSN  neuron  stands  out  with  its  unique  receptive  field,  with  a 
high  gain,  particularly  close  to  the  visual  midline  [38].  It  is  possible  that  the  uniquely  high  gain 
might  explain  the  strong  response  correlation  with  the  motion  of  a  salient  bar.  To  test  this,  we 
record  the  response  to  the  same  set  of  stimuli  in  HS  neurons  with  broader  receptive  fields.  To 
summarize  the  comparison  we  measure  the  absolute  mean  difference  during  the  400ms  the  bar 
passes  the  center  of  the  receptive  field.  We  use  the  absolute  difference  to  avoid  averaging  out 
positive  and  negative  differences,  as  in  the  red  (backgroundpref,  barpref)  and  the  green  data 
(backgroundpref,  barami-pref). 

We  found  that  male  HSN  responses  to  our  bar-background  stimuli  are  more  similar  to  bar 
motion  alone  (black.  Figure  14),  than  to  background  motion  alone  (grey.  Figure  14),  or  to  the 
sum  of  the  responses  to  the  individual  stimuli  (white.  Figure  14). 

Even  if  bigger  than  the  male  HSN  receptive  field,  the  female  HSN  also  has  a  small  receptive 
field,  sharply  delineated  at  the  mid-line  [38].  In  female  HSN  bar-background  responses  are  also 
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more  similar  to  bar  motion  alone  (blaek,  Figure  14),  than  to  background  motion  alone  (grey, 
Figure  14),  or  to  the  sum  of  the  responses  to  the  individual  stimuli  (white,  Figure  14). 
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Figure  14:  The  high  correlation  with  bar  responses  is  unique  to  HSN.  Comparison 
between  the  responses  to  flgure-ground  motion,  and  the  responses  to  bar  motion 
alone  (black),  background  motion  alone  (grey),  or  the  sum  of  the  responses  to  bar 
and  background  motion,  respectively  (white).  The  data  show  the  absolute  mean 
difference  for  the  four  combinations  of  bar  and  background  motion  shown  in  Figure 
13.  Stars  (*)  indicate  significant  difference  (p<0.05,  paired  t-tests)  between  errors. 

However,  in  male  and  female  HSNE,  with  a  more  LPTC-typical,  laterally  extended  receptive 
field,  the  response  to  our  bar-background  stimuli  are  equally  similar  to  the  responses  to  bar 
motion  alone  (black,  Figure  14),  background  motion  alone  (grey,  Figure  14),  and  the  sum  of  the 
responses  to  the  individual  stimuli  (white.  Figure  14).  This  suggests  that  the  strong  correlation 
we  see  between  bar-background  motion  and  bar  motion  in  HSN,  has  to  do  with  the  unique 
receptive  field  properties  of  the  most  dorsal  HS  neuron  in  hoverflies,  rather  than  be  ubiquitous 
feature  for  LPTCs.  This  is  supported  by  previous  work  showing  a  high  gain  particularly  to 
vertically  extended,  salient  features,  in  hoverfly  HSN  [41]  [42]. 

Some  of  these  findings  have  been  included  in  a  paper  that  has  been  resubmitted  to  the 
journal  PLoS  One,  and  the  remainder  will  be  covered  in  a  manuscript  that  is  presently  in 
production. 

3. 4. 3  Centrifugal  stationary  inhibited  flicker  excited  (cSIFE)  Neuron 

During  the  second  year  of  the  project  we  published  a  paper  providing  further  support  for  non- 
EMD  type  input  to  the  fly  lobula  plate.  The  cSIFE  (centrifugal  stationary  inhibited  flicker 
excited)  neuron  has  extremely  intriguing  physiological  response  properties.  It  responds  with  a 
low  but  finite  spike  rate  to  a  visual  stimulus  with  no  spatial  contrast,  but  if  a  high-contrast, 
stationary  pattern  appears  on  the  screen,  it  gives  one  or  two  transient  spikes  followed  by 
pronounced  inhibition  of  the  spike  rate  (Figure  15 A,  C).  In  contrast,  the  cell  is  excited  -  i.e. 
displays  significant  increase  in  spike  rate  over  the  no-contrast  case  -  if  the  pattern  moves  (Figure 
15B,  D)  or  flickers.  During  the  third  year,  we  continued  characterization  of  cSIFE. 
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In  our  Year  2  work,  we  found  that  eSIFE  was  strongly  excited  by  flicker,  up  to  high  temporal 
frequencies,  with  ON  and  OFF  contrast  changes  coded  independently.  The  flicker  sensitivity 
leads  to  responses  to  stimuli  moving  in  any  direction  (Figure  15D).  Importantly,  however,  the 
strong  inhibition  of  cSIFE  by  stationary  sinusoidal  gratings  of  any  orientation  (Figure  15C)  is 
unlike  flicker-sensitive  neurons  described  in  earlier  visual  pathways  [43].  cSIFE  could  thereby 
serve  as  a  neural  substrate  for  the  coding  of  stationary  stimuli,  as  well  as  of  high-frequency 
flicker,  such  as  might  be  associated  with  sudden  perturbations  from  visual  stability. 

cSIFE  is  a  heterolateral  neuron  with  output  dendrites  covering  the  expanse  of  the  lobula  plate 
(Figure  15E).  These  outputs  affect  the  response  properties  of  neurons  sensitive  to  optic  flow  [15], 
and  cSIFE  may  thereby  play  a  profound  but  hitherto  unappreciated  role  in  the  coding  of  vision. 
Despite  having  its  output  dendrites  in  the  lobula  plate  (Figure  15E),  it  clearly  does  not  derive  its 
sensitivity  from  underlying  EMDs.  This  thus  provides  even  stronger  evidence  for  non-EMD  type 
input  to  the  lobula  plate  [13]. 


Figure  15:  Centrifugal  stationary  inhibited  flicker  excited,  a  novel  lobula  plate 
neuron.  A.  Neural  response  to  a  stationary  sinusoidal  grating  (0.1  cycles/degree) 
presented  for  Is.  B.  Neural  response  to  a  moving  sinusoidal  grating  (0.1 
cycles/degree,  5Hz).  The  bar  under  the  raw  data  indicates  the  stimulus  duration  for 
panels  A  and  B.  C.  Raster  plots  where  each  vertical  line  indicates  the  timing  of  an 
action  potential.  The  second  to  last  row  represents  the  data  shown  in  panel  A.  The 
data  in  each  row  show  the  response  to  a  stationary  grating  (0.1  cycles/degree)  of 
different  orientation,  as  indicated  by  the  pictograms  to  the  left.  D.  Raster  plots  of 
responses  to  sinusoidal  gratings  (0.1  cycles/degree,  5  Hz)  moving  in  different 
directions,  as  indicated  by  the  arrowheads  to  the  left.  The  second  row  from  the  top 
represents  the  data  shown  in  panel  B.  E.  Morphology  of  cSIFE.  The  neuron  was 
recorded  and  filled  in  the  left  lobula  plate.  Arrowheads  1  and  2  indicate  a  posterior 
dip  of  the  main  axon,  as  do  arrowheads  3  and  4.  Med  =  medulla,  LP  =  lobula  plate, 
Prot  =  protocerebrum. 


cSIFE’s  inhibition  by  spontaneous  patterns  does  not  depend  on  pattern  orientation  (Figure 
15C),  and  neither  does  the  excitation  depend  on  the  direction  of  motion  (Figure  15D).  Since 
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cSlFE’s  response  to  motion  is  non-directional,  whereas  directionality  is  a  key  feature  of  EMD- 
type  input,  we  went  on  to  study  its  dependence  on  pattern  wavelength.  We  found  that  the 
excitation  increased  with  pattern  wavelength  (Figure  17A),  also  arguing  against  an  underlying 
EMD-type  input. 

When  using  higher  temporal  frequencies  the  response  continued  to  increase  with  no  clear 
peak.  This  prompted  us  to  use  a  sinusoidal  flicker,  and  indeed,  the  response  was  stronger  than  to 
motion  (Figure  16,  green).  This  further  supports  the  notion  that  the  cSlFE  response  that  we 
recorded  to  motion  is  derived  from  a  high  sensitivity  to  flicker,  and  not  to  elementary  motion. 


Figure  16:  cSIFE’s  motion  response  is  flicker  derived.  The  response  of  cSIFE  as  a  function 
of  temporal  frequency  of  a  full-screen,  full-contrast  sinusoidal  grating  (0.1  cycles/degree, 
red,  N=ll).  The  red  data  show  the  response  to  motion,  and  the  green  to  full-screen, 
sinusoidal  flicker  at  a  subset  of  these  frequencies  (N=8,  except  at  2  and  40  Hz  where  N=6). 

Finally,  cSIFE  inhibition  by  stationary  patterns  is  limited  to  a  narrow  band  of  wavelengths 
(ca.  0.08-1  degrees).  When  displaying  patterns  with  wavelengths  above  or  below  this  cut-off,  we 
saw  no  inhibition  below  spontaneous  rates  (grey.  Figure  17A). 
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Spatial  frequency  (cpd) 


Figure  17:  Spatial  frequency  sensitivity.  A.  The  response  of  cSIFE  as  a  function  of 
spatial  frequency  of  a  stationary  full-screen,  full-contrast  sinusoidal  grating  (N=8). 
B.  The  response  of  cSIFE  as  a  function  of  spatial  frequency  of  a  full-screen,  full- 
contrast  sinusoidal  grating  moving  at  5  Hz.  N=10. 


These  findings  were  published  in  Journal  of  Neuroscience  [15],  with  some  key  points 
reproduced  here. 

During  the  final  year  of  the  project,  we  investigated  the  responses  of  cSlFE  to  natural 
images.  These  were  presented  either  as  stationary  (to  induce  inhibition),  or  moving  (to  induce 
excitation).  Here  we  present  the  inhibition  experiments.  We  chose  five  HDR  images  taken  from 
natural  hoverfly  habitats  [11],  and  the  broadband  image  used  in  the  experiments  above  (Figure 
12).  The  images  (Figure  18)  cover  a  range  of  contrasts  and  contain  differences  in  their 
distribution  of  features. 


Figure  18:  The  naturalistic  images  used  in  the  cSIFE  experiments.  Images  A-E  come 
from  a  database  of  HDR  images  obtained  by  Brinkworth  and  O’Carroll  from 
natural  hoverfly  habitats.  Image  F  has  been  used  in  several  publications,  including 
the  study  described  above  (Figure  12). 
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We  found  that  all  6  images  inhibited  the  spontaneous  rate  of  cSlFE,  but  that  the  inhibition  by 
the  different  images  varied  (Figure  19),  and  in  no  case  was  the  inhibition  as  strong  as  when  using 
single  frequency  sinusoidal  gratings  (Figure  17). 

To  investigate  this  in  more  detail,  we  plot  the  same  data,  but  use  the  image  contrast  on  the  x- 
axis.  This  analysis  shows  that  there  is  a  clear  correlation  between  image  contrast  and  the  strength 
of  the  inhibition.  This  compares  well  with  our  findings  from  our  Journal  of  Neuroscience  paper 
(using  sinusoidal  gratings). 
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Figure  19:  Inhibition  of  cSIFE’s  spontaneous  rate  by  different  natural  images 
(N=16).  The  plot  on  the  left  show  the  images  organized  as  described  in  Figure  18 
above.  The  plot  on  the  right  shows  the  inhibition  by  the  different  images  when  these 
are  organized  after  their  respective  contrast. 

As  of  the  close  of  the  project,  we  were  conducting  experiments  in  which  the  contrast  of  two 
of  the  images  (A  and  D)  was  manipulated  to  determine  how  that  affects  the  inhibition.  Results 
have  now  been  submitted  to  Nature  communications  and  are  presently  under  review. 

3.5  In-Vivo  Imaging 

During  the  final  reporting  period,  we  used  two-photon  excitation  calcium  imaging  in 
conjunction  with  genetic  techniques  to  target  and  examine  feature  detection  by  specific  cell  types 
in  vivo  to  various  types  of  figure  and  edge-related  stimuli.  We  survey  four  cell  types  that  project 
from  the  fly  lobula  -  a  proposed  feature  selective  region,  and  an  attractive  target  for  study  due  to 
the  paucity  of  physiological  work  that  has  been  done  there  -  and  which  terminate  within  discrete 
optic  glomeruli  in  the  central  brain.  We  find  that  the  physiological  responses  of  each  cell  type 
encode  different  features  of  figure-ground  stimuli,  composing  a  hierarchy  of  figure-like  percepts. 
Two  of  the  cell  types  we  probed,  termed  LC12  and  LClOa,  (Figure  20)  -  both  having  projections 
that  pass  through  the  Anterior  Optic  Tract  (AOT)  -  specifically  encode  sufficient  information  to 
discriminate  a  small  moving  figure  on  a  moving  ground,  while  showing  homologous  responses 
across  the  space  of  figure-like  stimuli  that  elicit  similar  behavioral  responses.  In  brief,  we  take 
advantage  of  a  yeast  transcription  factor,  Gal4,  which  labels  select  subsets  of  neurons.  We  use 
Gal4  to  “drive”  the  expression  of  any  gene  of  choice,  such  as  green  fluorescent  protein  (GFP)  for 
anatomical  identification,  or  GCaMP  for  in  vivo  calcium  imaging. 
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Figure  20:  Anatomy  of  two  example  lobula  glomerular  projection  cell  systems.  Indicated  are 
images  of  the  right  brain  hemisphere  with  prominent  optic  lobes,  lobula  (Lo),  medulla  (Me) 
and  the  ventrolateral  protorecerebrum  of  the  central  brain  (VLPR).  (A)  GFP  staining 
(green)  of  a  Gal4  line  (R65B05)  labeling  the  identified  LC12  columnar  neuron  in  the  lobula 
(Lo).  Inset  shows  a  single  cell  clone  isolated  from  the  full  expression  profile,  indicating 
dendrites  within  the  outer  layer  of  the  lobula  (near  the  lower  medulla),  with  an  axon 
projecting  into  a  glomerulus  of  the  ventrolateral  protocerebrum  (VLPR).  (B)  The  full  Gal4 
expression  profile  of  LClOa  columnar  cell  (R80G09  Gal4  line)  that  projects  through  the 
anterior  optic  tract  (AOT)  into  the  optic  tubercle  (OpTu).  Note  that  the  LClOa  line  labels 
extraneous  neurons  of  the  medulla  and  lobula  plate,  but  imaging  was  restricted  to  the  lobula, 
where  only  this  cell  class  is  labeled  by  the  Gal4  driver. 


The  four  anatomically  identifiable  cell-types  that  we  selected  for  study  are  distinguishable 
from  one  another  by  each  having  a  projection  pattern  to  a  distinct  optic  glomerulus  [51] [44]  or 
the  lobula  plate  [30].  Two  of  these  are  shown  here  for  example  (Figure  14).  The  lobula  is  the 
most  distal  area  of  the  optic  lobe  where  retinopic  inputs  are  broadly  recombined  and  outputs 
projected  into  glomeruli,  in  a  convergent  architecture  similar  to  that  seen  in  the  antenna  lobe. 
This  anatomical  convergence  suggests  that  the  lobula  forms  a  nexus  between  sensation  and 
abstract  feature  coding  [52][53].  LC12  is  a  columnar-type  neuron  that  projects  to  the  lateral 
portion  of  the  ventrolateral  protocerebrum  (VLPR).  LCIO  is  columnar-type  neuron  that  projects 
through  the  Anterior  Optic  Tract  (AOT)  to  the  anterior  portion  of  the  optic  tubercle  (optu).  These 
two  cell  classes,  plus  the  two  others  not  shown,  were  probed  in  vivo  with  a  suite  of  figure  and 
edge  stimuli;  results  from  such  experiments  on  LC12  and  LClOa  are  depicted  below  in  Figure 
21: 
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Figure  21:  Calcium  responses  from  lobula  projection  neurons  imaged  with  genetically 
encoded  fluorescent  calcium  reporter  GCaMP6.  A  live  fly  is  restrained  under  the  objective 
of  a  two-photon  scanning  laser  excitation  microscope,  viewing  a  LED  display  identical  to 
those  used  in  the  behavioral  experiments.  A  small  window  is  cut  in  the  rear  of  the  head 
capsule  to  view  the  Gal4-GCaMP  labeling  within  identiHed  neurons  of  the  lobula.  (A)  A  full- 
field  image  of  many  individual  columnar  LC12  neuron  dendrites  within  the  lobula.  False 
color  is  added  to  indicate  the  retinotopic  receptive  field  of  the  dendrites.  (B)  Excitatory 
calcium  responses  from  multiple  regions  of  interest  (ROIs)  ordered  from  the  nasal  (n)  to 
temporal  (t)  direction.  Upper  graph  shows  the  fluorescence  response  to  a  30-degree  Fourier 
bar  sweeping  across  the  LED  display.  Note  the  timing  offset  generated  by  the  sequential 
activation  of  ROIs  at  each  retinotopic  position.  We  observed  no  heterogeneity  in  the 
responses,  so  we  time-shifted  the  response  and  pooled  them  (lower  inset  panel).  (C)  Mean 
AF/F  responses  to  bar  motion  compared  to  switching  the  arena  ON  and  OFF.  (D,  E) 
Exemplar  responses  of  LClOa  and  LC12  to  complex  figure-ground  stimuli.  Insets  represent 
space-time  graphs  indicating  the  stimulus  conditions  (x-axis  =  visual  azimuth,  y-axis  =  time). 


For  imaging  responses,  flies  were  shown  a  perspeetive-matehed  version  of  the  stimulus  from 
our  behavioral  assays.  The  imaging  region  eontained  many  loeal  dendritie  proeesses  belonging  to 
multiple  individual  eolumnar  neurons  (indieated  in  Figure  21  A).  To  test  for  neuronal  responses 
that  eould  underlie  the  figure-pereeptual  eomputations  that  we  know  flies  make,  we  first 
reeorded  responses  to  a  30-degree  bar  sweeping  aeross  the  azimuth  of  the  LED  display.  The  bar 
stimulus  aetivated  sequential  retinotopie  eolumnar  elements  as  the  bar  swept  aeross  the  azimuth 
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(indicated  in  Figure  21  A,  monochrome  image  with  pseudocolor  mask).  Within  the  image  frame, 
we  recorded  from  multiple  regions  of  interest  (ROIs),  which  are  arranged  in  order  along  the 
nasal-temporal  axis  (Figure  2 IB  top).  The  width  of  GCaMP  excitatory  response  corresponds 
with  the  width  of  the  bar  (roughly  30-degrees)  passing  through  the  receptive  field  of  the  ROI. 
We  did  not  identify  any  significant  response  variation  along  the  n-t  axis,  indicating  that  the 
ensemble  of  columnar  elements  responds  similarly  to  the  sweeping  bar.  We  therefore  collapsed 
the  spatial  dimension  in  order  to  pool  responses  across  ROIs  in  time  (Figure  2 IB  bottom).  All 
classes  of  lobula  projection  neurons  showed  similar  retinotopic  homogeneity. 

We  compared  the  mean  AF/F  response  to  switching  the  entire  LED  arena  ON  and  OFF 
(Figure  2 1C).  The  response  amplitudes  within  dendritic  arbors  to  ON  and  OFF  were  very  small 
in  each  cell  type,  indicating  that  lobula  projections  neurons  are  strongly  tuned  to  motion  and 
other  motion-derived  features  rather  than  flicker. 

We  presented  a  battery  of  16  stimuli,  only  a  few  of  which  are  highlighted  here  as  a  proof-of- 
concept.  LClOa  shows  no  difference  in  response  to  the  two  directions  of  Ground-alone  stimulus, 
but  shows  a  very  different  response  to  a  “receding  curtain”  stimulus  (Figure  2 ID  arrow).  This  is 
perhaps  unexpected  because  the  receding-curtain-type  stimuli  begins  as  a  wide-field  stimulus 
and  only  changes  when  the  discontinuity  of  the  edge  passes  through  the  visual  field  leaving 
behind  a  static  visual  scene.  This  demonstrates  that  LClOa  has  a  strong  predilection  for  the  non- 
£’M  components  of  the  stimulus. 

By  contrast,  LC12  shows  nearly  identical  responses  to  ground  motion  in  each  direction, 
while  the  two  directions  of  Fourier  bar  motion  elicit  markedly  different  responses  (Figure  2 IE). 
By  contrast,  the  Theta  bar  responses  are  nearly  identical  (Figure  2 IE  double  arrow).  In 
combination,  these  suggest  a  mechanism  whereby  the  contributions  of  both  FM-direction  and 
EM-direction  and  size-tuning  interact  in  a  multivariate  way.  Taken  together,  these  two  cell 
classes  demonstrate  properties  of  complex  EM  and  FM  coding  in  a  manner  similar  to  that  Lee 
and  Nordstrom  found  in  the  lobula  plate.  These  are  exciting  developments  because  collectively 
these  results  demonstrate  the  first  clear  evidence  for  the  underlying  neural  mechanism  of  higher- 
order  motion  coding  in  the  fly  visual  pathway. 

However,  the  examples  shown  demonstrate  the  difficulty  of  systematic  evaluation  of 
responses  within  a  particular  neural  sub-type  across  stimulus-conditions  to  produce  conceptual 
models  for  the  underlying  coding  filters  that  these  cells  implement.  Which  stimuli  match  the 
underlying  neural  filter  the  best?  How  will  we  know  when  we’ve  identified  the  receptive  field 
properties  of  cell  classes  that  by  definition  extract  abstract  representations  of  visual  features?  To 
address  this  shortcoming  in  a  methodological  manner,  we  have  undertaking  a  graph  theoretical 
covariance  analysis  that  considers  all  possible  pair-wise  comparisons  between  responses  to  our 
battery  of  16  stimuli  that  parameterize  the  figure-ground  space.  Stimuli  are  then  schematized  as 
nodes,  and  those  that  elicit  statistically  indistinguishable  responses  within  a  particular  cell-type 
are  connected  by  an  edge,  and  the  ensemble  of  tested  stimuli  can  then  be  viewed  as  a  graph.  This 
work  is  ongoing  and  is  under  production  for  publication  -  submission  is  planned  for  the  first 
week  of  December,  2014. 

3.6  Mechanisms  and  Models 

3. 6. 1  Context  established  by  initial  work 

It  is  clear  from  the  results  obtained  from  this  and  the  prior  AFOSR-sponsored  work  that  flies 
do  not  rely  (strictly)  on  outputs  from  classical  EMDs  to  detect  visual  figures,  as  previously 
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hypothesized  [18]  [46].  Rather,  they  display  eomputational  capabilities  previously  thought  to  be 
the  domain  of  vertebrate  nervous  systems,  in  that  they  are  able  to  respond  to  differences  in 
higher-order  spatiotemporal  statistics  between  figure  and  background.  In  their  initial  study  of  the 
electrophysiology  of  the  LPTC  HI,  in  which  they  first  noted  an  unexpected  sensitivity  to  higher- 
order  motion,  Quenzer  and  Zanker  [45]  postulated  that  the  canonical  EMD  might  still  be  able  to 
support  such  sensitivity  in  conjunction  with  additional  nonlinearities  or  more  complex 
architectures  (i.e.,  rectification  of  the  inputs  to  HMDs,  and/or  multiple  cascaded  EMD  units). 
However,  although  simulations  were  run  and  the  results  presented  graphically,  quantitative  data 
were  not  given  to  support  this  hypothesis. 

The  electrophysiological  results  from  HS  and  VS  cells  documented  in  our  PNAS  paper  [32] 
are  fascinating  but  raise  as  many  questions  as  they  answer.  Sensitivity  to  higher-order  figure 
motion  is  associated  with  large  variances  in  the  neural  responses,  and  it  seems  to  be  distributed 
piecemeal  between  the  different  parts  of  the  receptive  fields  and  on  different  phases  of  the  input 
stimuli.  This  suggests  that  the  quasilinearity  observed  at  the  behavioral  level  may  be  cobbled 
together  from  distinctly  nonlinear  parts.  But  overall,  the  nature  of  higher-order  sensitivity  in  the 
HS  and  VS  cells  is  itself  uncertain:  are  these  neurons  primarily  integrators  of  EMDs  that  happen 
to  be  modulated  by  lateral  projections  from  dedicated  figure-detecting  cells,  or  are  all  LPTCs 
simply  integrators  of  varying  proportions  of  first-order  and  higher-order  motion  inputs? 

The  first  year’s  efforts  were  largely  consumed  with  characterizing  the  phenomenon  of 
higher-order  motion  sensitivity,  including  design  and  performance  of  experiments  and 
development  and  application  of  analytical  techniques  to  interpret  the  results.  Modeling  efforts 
were  confined  to  the  last  two  years  of  the  grant.  During  the  second  year,  we  developed  an 
explanatory  model  for  the  response  characteristics  of  the  novel  neuron  cSIFE,  and  in  addition 
developed  a  biophysical  model  for  the  correlator  function  that  is  thought  to  underlie  first-order 
motion  detection.  We  also  proposed  that  two  sequential  stages  of  computation,  presumably 
implemented  in  the  higher  optical  ganglia,  contribute  to  figure  sensitivity.  One  is  assumed  to 
comprise  some  fundamental,  local  set  of  neurocomputational  primitives,  possibly  including  (but 
not  necessarily  limited  to)  the  neural  analog  of  the  Reichardt  correlator,  and  a  second  constitutes 
some  means  to  compute  spatial  disparities  between  these  in  order  to  distinguish  figure  from 
background.  During  this  period,  we  followed  up  on  this  postulate,  and  examined  how  sets  of 
correlator  circuits,  operating  on  a  range  of  local  signals,  might  serve  as  primitives  for  the 
detection  of  spatiotemporal  differences  in  imagery  across  a  figure  boundary.  In  addition,  we 
modeled  a  neural  implementation  of  a  winner-takes-all  function  (which  is  not  directly  related  to 
neural  circuits  studied  so  far  during  the  project,  but  arose  out  of  Shoemaker’s  interest  in  NMD  A 
receptor-based  computation,  which  formed  the  basis  for  one  of  the  correlator  circuits  modeled  in 
Year  2). 

3. 6. 2  Modeling  of  cSIFE 

In  our  Journal  of  Neuroscience  paper  we  suggested  a  model  input  for  how  cSIFE  may  derive 
its  high  sensitivity  to  flicker,  and  strong  inhibition  by  stationary  patterns  (Figure  22).  In  the 
proposed  circuit  early  visual  input  is  separated  into  ON  and  OFF  luminance  changes,  which 
could  be  done  via  half-wave  rectification  in  LI  and  L2,  respectively  [4].  In  our  model  circuitry, 
the  ON  and  OFF  output  is  further  separated  into  an  inhibitory  (blue)  and  an  excitatory  (red) 
pathway  (Figure  22A). 

The  ON  and  OFF  inputs  are  processed  independently  via  temporal  high-pass  filters  (HP, 
Figure  22A),  leading  to  excitatory  responses  to  high-frequency,  high-contrast  luminance  changes 
(as  in  Figure  17).  Unlike  the  EMD,  there  is  no  spatial  correlation  between  neighboring  inputs  in 
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the  excitatory  pathway  (see  Figure  22A).  This  leads  to  larger  responses  to  full-screen  flicker  than 
to  drifting  gratings  with  the  same  temporal  frequency  (Figure  16),  and  greater  sensitivity  Figure 
15to  lower  spatial  frequencies  (Figure  17B),  since  these  generate  stronger  luminance  changes  at 
the  level  of  individual  photoreceptors.  Similarly,  the  lack  of  spatial  correlation  leads  to 
insensitivity  to  the  direction  of  motion  (Figure  15E). 

For  inhibition  by  stationary  patterns  the  ON  and  OFF  signals  are  first  processed  in  temporal 
low  pass  filters  (LP,  Figure  22A).  The  ON  signal  from  one  input  is  then  subtracted  from  the  OFF 
signal  from  a  spatially  separated  input  (‘-‘,  Figure  22A),  and  vice  versa  (not  shown,  for 
simplicity).  The  lateral  inhibition  between  inputs  of  opposite  contrasts  leads  to  inhibitory 
responses  to  stationary,  high-contrast  patterns  (Figure  15  A,  C). 

The  temporal  low  pass  filter  (LP,  Figure  22A)  suggests  that  extremely  slow-moving  patterns 
should  also  generate  inhibition.  Indeed,  the  response  to  a  0.5  Hz  sinusoidal  grating  is  inhibited 
compared  with  the  spontaneous  rate  (Figure  17 A,  N=4).  In  the  publication  we  performed  one 
direct  test  of  the  model  circuitry  (Figure  22B),  but  we  will  eventually  expand  on  this  set  of 
experiments  using  a  range  of  artificial  and  natural  images  that  we  are  manipulating  in  space  and 
time 


Figure  22:  Proposed  input  circuitry  to  cSIFE.  A.  The  diagram  shows  a  proposed  input 
circuitry  to  cSIFE.  Early  visual  processing  is  separated  into  ON  and  OFF  pathways,  here 
shown  as  half-wave  rectification.  The  ON  and  OFF  inpnts  are  then  either  subjected  to 
temporal  high-pass  filtering  (HP),  or  low-pass  filtering  (LP),  followed  by  subtraction  of  the 
opposite  contrast  from  a  neighboring  nnit  (-,  only  one  polarity  shown  here).  The  high-pass 
filtered  output  leads  to  excitatory  responses  in  cSIFE  (red),  whereas  the  subtracted  output 
leads  to  inhibition  (blue).  The  cSIFE  axon  crosses  the  visual  midline  where  it  may  interact 
with  other  LPTCs,  such  as  HSN.  B.  The  response  of  cSIFE  to  0-5  Hz  sinusoidal  gratings  (6° 
wavelength). 

3. 6. 3  Modeling  of  biophysically-plausible  ‘delay’  and  ‘correlate  ’  mechanisms 

The  Hassenstein-Reichardt  EMD,  based  fundamentally  on  a  ‘  delay- and-correlate’  operation, 
has  been  a  dominant  model  for  visual  motion  processing  for  over  50  years.  In  addition,  recent 
work  has  suggest  that  ‘delay-and-correlate’  operations  are  almost  certainly  crucial  to  the 
detection  of  small  visual  targets  [59],  and  we  hypothesize  that  they  may  also  play  a  role  in  figure 
detection.  However,  although  a  number  of  plausible  mechanisms  exist  for  a  neural  ‘delay’,  the 
way  in  which  a  ‘correlate’  operation  might  be  implemented  in  a  neural  substrate  remains  unclear. 
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We  have  eoneeived  a  basie  neural  ‘eorrelator’  that  depends  on  the  aetion  of  ehemieal 
synapses,  in  partieular  the  faets  that  synaptie  reeeptors  modulate  membrane  conductance  by  the 
opening  or  elosing  of  ion  ehannels,  and  that  various  elasses  of  ehannels  have  differing  reversal 
potentials  assoeiated  with  them.  Figure  23  shows  a  minimal  neural  model  in  whieh  a  eorrelation- 
like  operation  is  aehieved:  exeitatory  input  to  a  target  neuron  N1  is  supplied  by  input  ehannel 
CHI  and  its  effeetiveness  is  modulated  by  ehanges  in  the  incremental  membrane  resistance  of 
N1  that  are  driven  by  input  ehannel  CH2.  The  effeet  on  the  eireuit  output  of  CHI  aeting  alone  is 
nulled  out  in  neuron  N2. 


Figure  23:  Proposed  Model  for  a  neural  ^correlator’.  Signals  carried  in  two  input  channels 
CHI  and  CH2  interact.  Large  open  circles  represent  neurons,  open  triangles  excitatory 
synapses,  small  open  circles  hyperpolarizing  inhibitory  synapses,  and  small  closed  circles 
synapses  that  are  shunting  in  their  effect.  Diagram  A  depicts  the  basic  scheme.  The  essential 
nonlinear  interaction  takes  place  in  neuron  AT,  for  which  a  single-compartment  electrical 
model  is  shown  at  bottom.  In  this  circuit,  the  node  at  center  represents  the  cell  interior,  with 
instantaneous  potential  with  respect  to  the  extracellular  space  (Aground’).  The  passive  or 
unstimulated  membrane  is  characterized  by  resistance  resting  potential  and 

capacitance  Channel  CHI  activates  synapses  onto  N1  whose  ion  channels  have  an 
excitatory  reversal  potential  F/jg;  channel  CH2  inversely  modulates  synapses  with  an 
effective  reversal  potential  that  is  equivalent  to  or  very  near  the  resting  potential  Vjiy. 

This  modulation  could,  for  example,  represent  shunting  inhibition  that  is  reduced  by 
activation  of  CH2,  as  suggested  in  Diagram  B.  In  this  circuit,  N4  represents  an  interneuron 
that  imposes  tonic  shunting  inhibition  in  AT,  and  inhibition  of  N4  via  CH2  relieves  this  tonic 
inhibition.  The  neuron  N2  performs  a  nulling  function:  its  two  inputs  are  weighted  such  that, 
in  the  absence  of  activity  in  channel  CH2,  the  effect  of  excitatory  input  from  N1  is  nominally 
canceled  by  inhibition  via  interneuron  N3.  As  CH2  is  activated,  the  effectiveness  of  excitation 
of  N1  by  CHI,  and  consequently  of  N2  by  AT,  increases  in  a  graded  fashion,  and  the 
inhibition  in  N2  is  overcome. 

The  suggestion  that  modulation  of  membrane  resistance  in  this  circuit  might  be  implemented 
by  relief  of  shunting  inhibition  is  related  to  the  longstanding  notion  that  shunting  inhibition  could 
serve  as  a  mechanism  for  neural  gain  control  and/or  arithmetic  division.  One  issue  with  its 
application  to  a  multiplier  is  that  the  modulation  of  conductance  by  input  CH2  results  in  a 
supralinear  dependence  of  the  circuit  output  on  that  signal.  However,  this  characteristic  would  be 
counteracted  by  a  number  of  biophysical  phenomena  associated  with  neural  function  -  including 
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limits  on  vesicle  release  in  presynaptic  neurons,  limits  in  the  number  of  receptors  available 
postsynaptically,  and  saturation  of  membrane  potential  in  the  postsynaptic  neuron  -  that  tend  to 
be  sub-linear  or  compressive  in  their  effects.  Some  simulation  results  for  this  network  with 
single-compartment  neural  models  under  dc  conditions  are  shown  in  Figure  24. 


A  B 


Figure  24:  Parametric  multiplier  curves  for  the  neural  circuits  depicted  in  Figure  23.  The 
ordinate  is  the  membrane  potential  of  Nl^  the  abscissa  is  excitatory  synaptic  conductance  in 
N1  commanded  by  CHI,  expressed  in  terms  of  passive  or  unstimulated  membrane 
conductance  1/Ry.  Panel  A  pertains  to  the  3-neuron  circuit  of  Figure  lA;  the  parameter  is 
the  shunting  conductance  Ggh  in  that  is  decreased  by  activation  of  CH2  (also  expressed  in 
terms  of  1/Rr)>  It  is  varied  in  increments  of  1  unit.  The  inset  shows  the  small-signal  behavior 
of  the  circuit  with  respect  to  G^xc^  Panel  B  pertains  to  Figure  IB,  and  in  this  model  the  CH2 
input  is  subject  to  compressive  nonlinearities  in  neuron  N4,  which  model  saturation  of 
available  receptors  at  the  input  synapses  and  membrane  voltage  limiting.  The  parameter  is 
the  strength  of  the  inhibitory  input  imposed  on  N4  via  CH2,  varied  in  equal  increments.  The 
shunting  inhibitory  conductance  in  N1  that  is  driven  by  N4  is  scaled  so  that  it  assumes  the 
same  total  range  as  in  Panel  A.  The  resting,  shunting,  excitatory,  and  hyperpolarizing 
inhibitory  reversal  potential  values  are  set  to  =  — 60mV,  =  OV,  and  Vm  = 

— 90mV,  respectively,  and  apply  to  all  neurons. 

Of  further  interest  is  a  seeond  possible  biophysieal  meehanism  for  modulation  of  membrane 
resistanee  in  the  interaetion  neuron  Nl,  one  relies  on  the  inherently  nonlinear/nonmonotonie 
eleetrieal  properties  of  the  NMD  A  receptor  (NMDAR).  A  negative  slope  eonduetanee  regime 
assoeiated  with  NMDARs  ean  result  in  an  increase  in  ineremental  membrane  resistanee  when 
they  are  aetive  in  a  neuron  -  and  this  ean  give  rise  to  a  multiplieative  effeet  in  the  eontext  of  the 
‘eorrelation’  meehanism  proposed  herein,  when  the  input  ehannel  CH2  is  assoeiated  with 
NMDA  synapses.  Sueh  amplifying  effeets  also  typieally  require  eoaetivation  of  hyperpolarizing 
inhibitory  ehannels. 

DC  operating  eharaetisties  of  sueh  a  model  are  shown  below  in  Figure  25. 
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Figure  25:  Parametric  multiplier  curves  for  an  implementation  of  the  ‘mnltiplier’  network 
that  relies  on  the  nonmonotonic  current-voltage  characteristic  of  NMDARs.  The  ordinate  is 
the  membrane  potential  of  N2  and  the  abscissa  the  excitatory  synaptic  condnctance  in  N1 
commanded  by  CHI,  expressed  in  terms  of  passive  or  unstimulated  membrane  conductance 
X/Ry  Hyperpolarizing  inhibitory  inpnts  to  N1  are  assnmed  to  be  activated  in  proportion  to 
NMDAR  activation.  In  Panel  A,  the  parameter  is  a  measnre  of  the  slope  condnctance  GjvmdyI 
of  active  NMDARs  driven  via  CH2,  varied  in  equal  increments,  and  also  expressed  relative 
to  l/Rr*  In  panel  B,  satnration  of  the  nnmber  of  available  NMDARs  is  also  modeled. 
NMDAR  condnctance  is  scaled  so  that  it  assnmes  the  same  total  range  as  in  Panel  A. 

One  issue  with  the  NMDAR-based  model  is  that  co-aetivation  of  hyperpolarizing  membrane 
conductance  must  be  made  in  strict  proportion  to  NMDAR  activation,  to  prevent  offsets  in  the 
operating  curves.  It  is  difficult  to  conceive  of  a  mechanism  involving  inhibitory  intemeurons  that 
can  maintain  the  required  degree  of  linearity.  However,  a  possible  intrinsic  mechanism  to 
achieve  this  might  be  the  presence  of  SK  channels  in  the  membrane  of  N1 :  channels  of  this  type 
are  activated  in  proportion  to  the  concentration  of  calcium  conducted  into  the  neuron  by  active 
NMDARs.  SK  channels  and  NMDARs  are  in  fact  found  in  close  association  in  vertebrate 
cortical  neurons,  and  the  existence  of  just  such  a  feedback  mechanism  has  been  demonstrated  in 
that  context  [40].  The  hyperpolarizing  current  conducted  by  SK  channels  necessarily  lags 
NMDAR  activation  in  time,  and  thus  activation  of  CH2  in  the  model  alone  would  cause  transient 
depolarization  in  the  model  -  but  such  transients  are  consistent  with  startup  transients  observed 
in  motion-sensitive  and  other  higher  visual  neurons  in  the  insect  visual  pathway.  This  concept 
shows  promise  in  ongoing  simulations. 

Finally,  we  are  examining  the  possibility  that  the  kinetics  of  synaptic  receptors  in  the  CH2 
signal  path  might  comprise  a  significant  component  of  the  delay  operator  in  a  ‘delay-and- 
correlate’  circuit  in  insect  vision.  NMDARs  have  very  slow  kinetics  for  an  ionotropic  receptor, 
and  GABAb  receptors  -  which  are  a  likely  mechanism  for  shunting  inhibition  in  the  original 
circuit  of  Figure  23  -  are  metabotropic  and  even  slower.  Simulation  of  the  temporal  frequency 
response  of  an  elementary  motion  detector  using  linearized  models  for  vertebrate  receptors  of 
these  two  types  show  optima  at  frequencies  below  (although  within  an  order  of  magnitude  of) 
similar  curves  derived  from  motion-sensitive  neurons  in  insect  brains,  as  suggested  in  Figure  26. 
(It  should  be  noted  that  these  optima  are  similar  to  those  seen  in  crepuscular  and  nocturnal 
insects.)  Both  receptor  types  are  subject  to  the  effects  of  allosteric  neuromodulators,  which  could 
provide  a  mechanism  for  (real-time)  modulation  of  the  tuning  or  gain  of  HMDs. 
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Figure  26:  Temporal  frequency  tuning  of  Hassenstein-Reichardt  EMD  with  delay  operators 
based  on  receptor  kinetics.  The  ordinate  is  EMD  output  normalized  by  its  maximum  value 
for  each  curve.  The  gray  curve  is  for  GABAb  and  black  for  NMDA  receptors.  There  is  no 
prefiltering  or  prior  nonlinearity  used  in  the  model. 

These  findings  were  presented  in  a  poster  entitled  “A  biophysical  model  for  neural  ‘delay- 
and-correlate’”  at  the  ICIV  conference  in  2013. 

3.6.4  Fundamental  computations  underlying  figure  detection 

As  noted  in  the  introduction,  during  the  course  of  the  project  we  introduced  the  hypothesis 
that  correlations  or  product-like  operations  computed  pairwise  over  a  variety  of  different  visual 
signals  might  provide  a  primitive  basis  for  detection  of  the  presence  of  a  figure  (or  figure 
boundary).  In  the  final  year  of  the  project,  we  have  initiated  a  study  to  test  this  hypothesis. 

The  inspiration  for  this  approach  comes  from  models  for  motion  detection  functions  in  the 
insect  visual  system.  The  well-known  Hassenstein-Reichardt  correlator  [24]  (although  its 
biophysical  substrate  remains  unknown)  has  been  accepted  for  years  as  a  model  for  local  motion 
detection,  and  is  believed  to  constitute  the  computational  primitive  that  underlies  analysis  of 
wide-field  motion  [25]  [36]  [50].  This  model  involves  a  pairwise,  product-like  operation  between 
delayed  local  and  undelayed  neighboring  visual  signals  (or  vice-versa).  Recent  work  suggests 
that  these  are  derived  from  separate  ON  and  OFF  channels  that  are  segregated  earlier  in  the 
visual  pathway  [30].  The  Reichardt  detector,  however,  is  not  the  only  form  of  motion  detection 
that  has  been  modeled  with  a  ‘correlator’.  Research  on  small  target  motion  detecting  neurons 
(STMDs)  provides  evidence  that  a  product-like  operation  also  underlies  the  sensitivity  to  small 
dark  targets  that  is  observed  in  these  cells  [59] [60]  -  although  in  this  case,  the  ‘correlation’ 
involves  local  delayed  OFF  vs.  local  undelayed  ON  signals,  each  with  highly  transient  response 
characteristics. 

This  has  led  us  to  postulate  that  such  product-like  operations  might  act  as  computational 
primitives  for  more  general  forms  of  spatiotemporal  analysis,  including  the  detection  of  figures 
or  figure  boundaries.  With  this  in  mind,  we  began  by  considering  a  range  of  signals  known  or 
believed  to  be  present  in  each  visual  unit  of  the  lamina  and/or  medulla,  which  might  comprise 
the  inputs  for  an  expanded  set  of  correlators  that  could  serve  this  purpose.  These  include  the 
aforementioned  ON  and  OFF  channels,  with  additional  characteristics  including  transient  [29]  or 
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sustained  [28]  temporal  responses,  and  delayed  versus  undelayed  signal  pathways.  A  schematic 
diagram  illustrating  the  concept  is  shown  below  in  Figure  27: 
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Figure  27:  Schema  for  spatiotemporal  processing  that  is  hypothesized  to  support 
figure  detection.  Signals  from  photoreceptors  are  segregated  into  ON  and  OFF 
channels,  with  the  former  responding  to  increments  in  luminance,  and  the  latter  to 
decrements.  These  are  further  segregated  into  channels  with  transient  or  sustained 
response  characteristics  (with  the  sustained  channels  emphasizing  high-frequency 
components  but  having  a  finite  dc  response).  Each  of  these  may  be  routed  by 
delayed  or  undelayed  pathways  to  a  set  of  ‘correlators’,  presumed  to  reside  in  the 
medulla  and  each  represented  by  a  circle  with  an  ‘X’  at  its  center.  At  bottom  left  is 
shown  a  hypothetical,  ‘neuro-plausible’  unit  for  spatiotemporal  discrimination, 
which  consists  of  a  weighted  sum  of  correlator  outputs  cascaded  with  a  thresholding 
operation. 

We  considered  signals  with  every  possible  combination  of  these  attributes,  leading  to  a  set  of 
eight  per  visual  unit.  For  generality,  we  also  considered  every  possible  product  formed  between 
the  set  of  signals  that  are  either  internal  to  a  visual  processing  unit  or  arise  from  its  nearest- 
neighboring  units.  This  leads  to  28  possible  internal  products  (neglecting  the  simple  square  terms 
that  arise  from  multiplying  a  signal  by  itself),  and  64  products  with  signals  from  each  neighbor. 
These  are  assumed  to  form  the  basis  for  the  discriminations  that  allow  the  system  to  detect 
different  spatiotemporal  statistics  across  a  figure  boundary. 

We  propose  a  simple,  ‘neuro-plausible’  model  for  a  unit  that  could  perform  such 
discriminations  in  the  form  of  a  weighted  summation  (corresponding  to  the  action  of  weighted 
input  synapses)  of  outputs  from  some  subset  of  correlator  units,  followed  by  a  thresholding 
operation.  With  a  ‘hard’  threshold  and  binary  output,  such  a  unit  could  be  interpreted  as 
estimating  a  strict  class  membership  function:  it  is  intended  to  be  active  when  the  spatiotemporal 
characteristics  of  its  inputs  fall  within  a  certain  range,  and  inactive  otherwise.  One  such  unit  is 
depicted  at  bottom  left  in  Figure  27.  It  is  assumed  that  such  units  would  be  repeated 
retinotopically,  each  having  a  consistent,  spatially-shifted  receptive  field  (corresponding  to  the 
projections  from  its  corresponding  set  of  correlators).  It  is  also  supposed  that  multiple  classes  of 
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such  units  might  exist,  each  ‘tuned’  to  discriminate  according  to  some  particular  criterion  that 
would  be  determined  by  its  individual  weighting  pattern. 

In  preliminary  work  on  these  concepts,  we  developed  a  numerical  model  for  the  network 
architecture  suggested  in  Figure  27.  The  photoreceptor  is  represented  with  an  adaptive  Naka- 
Rushton  equation  [37]  (adaptation  time  constant  x  =  Is),  and  ON/OFF  segregation  by  a  first- 
order  highpass  filter  (x  =  200ms)  followed  by  half-wave  rectification  for  each  phase  (with  the 
OFF  phase  inverted  so  that  both  channels  carry  positive  signals).  The  ‘sustained’  temporal 
characteristic  is  modeled  with  a  ‘relaxed’  linear  highpass  filter  (x  =  50ms)  that  passes  40%  dc. 
The  ‘transient’  characteristic  is  based  on  the  nonlinear  transient  cell  model  of  Wiederman, 
Shoemaker,  and  O’Carroll  [60].  The  delay  operator  is  a  first-order  lowpass  filter  (x  =  50ms).  The 
‘correlators’  are  modeled  as  simple  products. 

The  discrimination  function  is  modeled  as  two-state  vector  quantization  on  the  set  of 
correlator  signals  that  form  the  ‘receptive  field’  of  the  discriminator  unit.  An  input  vector  is 
associated  with  one  class  or  the  other  according  to  which  of  two  prototypes  it  is  closest  to  by 
Euclidean  distance.  It  is  straightforward  to  show  that  this  geometrically-defined  procedure  yields 
the  same  result  as  a  weighted  summation  and  thresholding,  where  the  weighted  sum  is  the  inner 
product  of  the  input  vector  and  the  (normalized)  difference  vector  between  the  two  prototypes, 
and  the  threshold  is  equal  to  the  projection  of  the  midpoint  between  the  prototypes  onto  an  axis 
parallel  to  the  difference  vector  and  passing  through  the  origin.  (Naturally  this  holds  for  any 
weight  set  and  threshold  proportional  to  these  values.)  This  formulation  allows  us  to  use 
clustering  algorithms  to  determine  an  optimal  or  near-optimal  weight  set  over  some  ensemble  of 
inputs  that  can  be  divided  a  priori  into  two  classes  (e.g.,  a  class  that  originates  within  a  figure 
and  a  second  that  originates  without).  Presumably,  if  a  similar  scheme  for  discrimination  were 
present  in  the  fly,  then  evolutionary  pressures  would  likewise  have  driven  the  synaptic  weights 
toward  optimal  values.  To  optimize  the  prototype  vectors  /  weights,  a  multi-start  iterative 
procedure  was  used.  This  implemented  the  k-means  clustering  algorithm  (with  k=2)  when  it 
resulted  in  improvement  in  classification  at  a  particular  iteration,  and  a  heuristic  method  of 
prototype  adjustment  when  it  did  not.  (As  is  typical  with  nonlinear  optimization,  there  is  no 
guarantee  of  finding  a  global  maximum  with  this  method,  but  multiple  trials  suggest  that  it  can 
come  close.) 

An  initial  set  of  simulations  has  been  run  to  test  this  model  with  a  small  network  containing  a 
one-dimensional  array  of  eight  ‘photoreceptors’  and  their  accompanying  visual  processing  units. 
Correlations  were  formed  between  the  internal  signals,  and  with  the  neighboring  signals,  for  the 
central  six  units  in  this  array.  Sequences  of  time-varying  imagery  were  generated  to  form  the 
input  data  for  these  simulations.  To  date,  these  have  been  derived  from  photographic  data  of 
natural  imagery,  so  that  spatial  statistics  are  similar  to  those  that  might  be  encountered  by  an 
animal  in  its  environment.  In  each  series  of  input  data,  there  is  a  sharp  boundary  at  the  center  of 
the  image  with  different  spatiotemporal  characteristics  in  each  half  -  for  example,  motion  on  the 
left  and  flicker  on  the  right  -  and  no  correlation  across  the  boundary  (i.e.,  the  halves  are  derived 
from  different  parts  of  a  panoramic  master  image).  To  form  the  inputs  to  the  model  network,  the 
image  sequences  are  downsampled  onto  a  1  x  8  grid  with  a  spatial  separation  of  1.25°,  and 
blurred  by  spatial  convolution  with  a  Gaussian  kernel  that  has  a  half-maximum  width  of  1.4 
times  this  value,  in  order  to  mimic  the  antialiasing  properties  of  compound  eye  optics.  These 
parameters  yield  ‘retinal’  images  with  resolution  similar  to  that  in  larger  fly  species  (such  as 
Eristalis).  The  (blurred)  boundary  at  the  center  of  the  image  defines  a  stationary  ‘figure  edge’ 
that  is  located  between  the  left  and  right  sides  of  the  model  array.  As  suggested  by  work  in  the 


39 


FA9550-12-1-0034  Final  Report 


30  MAY  2014 


Frye  lab  reported  herein,  such  edges  or  boundaries  appear  to  be  the  key  features  that  are 
exploited  for  figure  detection  in  the  fly. 

Initially,  five  classes  of  figure  boundaries  were  generated,  with  the  following  attributes  on 
the  two  sides  of  the  array: 

-  Static  pattern  vs.  coherent  motion 

-  Static  pattern  vs.  flicker 

-  Motion  vs.  flicker 

-  Motion  left  vs.  motion  right 

-  Flicker  vs.  high-frequency  flicker 

It  soon  became  apparent  that  the  scenarios  with  static  patterns  were  trivial  cases  as  far 
discrimination,  due  to  the  lack  of  temporal  variation  in  the  downstream  visual  signals. 
Simulations  focused  on  the  final  three  scenarios,  and  data  were  generated  from  five  different 
natural  images  for  each,  with  35,000  total  samples  generated  at  10ms  sample  intervals.  Output 
data  were  sampled  at  the  same  rate.  Moving  imagery  was  animated  at  50°/s,  and  flicker  at  the 
lowest  frequency  was  obtained  by  animation  perpendicular  to  the  array  axis  at  the  same  speed. 
Flicker  at  five  times  this  frequency  was  obtained  by  dithering  randomly  between  different 
regions  of  the  master  image  at  an  appropriate  rate,  with  smoothing  between  image  changes. 

Vector  quantization  into  one  of  two  classes  was  performed  and  optimized  based  on  the 
outputs  of  the  correlators  in  the  array,  independently  for  each  scenario.  This  is  equivalent  to 
assuming  the  existence  of  a  distinct  class  of  optimized  ‘neural  discriminator’  for  each. 
Furthermore,  the  correlator  outputs  were  arranged  in  different  groupings  corresponding  to 
different  possible  ‘receptive  fields’  for  the  discriminators.  One  grouping  included  all  correlations 
with  at  least  one  factor  arising  from  a  particular  visual  processing  unit  (i.e.,  28  internal  signals 
and  64  in  association  with  each  neighbor,  for  a  total  of  156),  and  another  only  the  correlations 
between  neighbors  (64  signals).  Note  that  discriminators  with  the  first  grouping  have 
overlapping  receptive  fields. 

Discrimination  capability  was  evaluated  and  scored  according  to  what  fraction  of  the  outputs 
from  each  of  the  two  classes  was  grouped  together,  evaluated  over  both  classes  and  all  samples 
for  a  given  input  scenario.  (Note  that  purely  random  class  assignment  in  this  two-class  problem 
would  result  in  a  score  of  50%.)  For  the  unit-wise  receptive  fields,  the  two  classes  corresponds 
simply  to  the  location  of  a  unit  in  the  left  or  the  right  half  of  the  image.  However,  for  the  smaller 
inter-unit  receptive  fields,  in  addition  to  left-right  partitioning,  the  correlations  across  the  figure 
boundary  were  compared  to  correlations  originating  entirely  from  within  either  the  left  or  right 
side.  Figure  28  below  illustrates  discrimination  performance: 
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‘Receptive  Field’  configuration 


Score  (%  assigned  to  ‘correct’ class) 
Motion|Motion  Motion|Flicker  Flicker|Flicker 

84.5%  76.6%  90.0% 

92.6%  75.0%  92.8% 

83.0%  74.4%  91.3% 

96.0%  98.6%  96.9% 

95.4%  93.6%  95.4% 


Figure  28:  Scoring  of  discrimination  of  differing  spatiotemporal  statistics  according 
to  the  scheme  described  in  the  text.  Inputs  with  differing  spatiotemporal  statistics 
are  presented  to  the  right  and  left  halves  of  the  visual  processing  array,  as  indicated 
by  the  headers  in  the  three  columns  of  data  at  right.  In  the  ideograms  at  left,  visual 
processing  units  are  indicated  by  gray  circles,  internal  correlations  by  the 
intersecting  lines  under  each  circle,  and  inter-unit  correlations  by  intersecting  lines 
between  them.  Location-based  partitioning  of  discriminators  into  two  classes  is 
indicated  by  different  colored  brackets  that  extend  over  the  receptive  fields  of  each 
class.  Scores  in  each  row  indicate  the  percentage  of  the  time  (i.e.,  the  fraction  of 
samples)  that  the  discriminator  outputs  for  one  class  are  the  same  and  in  the 
opposite  state  from  those  of  the  other  class,  for  each  of  three  input  scenarios.  The 
top  two  rows  show  results  for  unit-wise  receptive  fields,  and  the  bottom  three  rows, 
for  inter-unit  receptive  fields.  In  the  scoring  for  the  bottom  two  rows,  each 
discriminator  on  the  side  (receptive  fields  in  green  or  red)  is  compared  individually 
to  the  central  discriminator,  and  the  score  compiled  over  all  three  comparisons. 
Significance  of  the  results  is  discussed  in  the  text. 

In  the  results  shown  in  Figure  28,  note  that  an  independent  optimization  has  been  performed 
for  each  input  scenario  indicated  in  the  columns  at  right,  and  for  the  receptive  field 
configurations  shown  in  each  row.  For  the  unit-wise  receptive  fields  (top  two  rows),  when  the 
central  discriminators  (which  both  receive  inputs  from  the  cross-boundary  correlators)  are 
excluded,  the  result  is  a  modest  improvement  in  performance  for  two  of  the  three  input  scenarios. 
When  inter-unit  receptive  fields  that  arise  exclusively  from  units  on  one  side  or  the  other  (third 
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row)  are  considered,  scores  are  similar  to  those  for  the  unit-wise  receptive  fields.  However,  when 
the  receptive  field  that  extends  across  the  figure  boundary  (coded  in  orange  in  rows  4  and  5)  is 
compared  to  those  arising  from  one  side  or  the  other,  the  discrimination  scores  are  consistently 
higher  than  for  the  other  cases,  being  above  90%  in  all  cases. 

A  single  important  lesson  can  be  drawn  from  these  results:  in  a  correlation-based  scheme  for 
spatiotemporal  discrimination  to  support  figure  detection,  it  is  the  correlations  across  figure 
boundaries  that  appear  to  carry  the  most  information  with  respect  to  discrimination. 

The  vector  quantizations  described  above  are  based  on  a  very  high-dimensional  vector  space, 
i.e.,  that  of  every  possible  product  of  signal  pairs  of  the  various  assumed  types.  Although  each 
medullar  column  contains  an  impressive  amount  of  neural  machinery  (>350  individual  cells  or 
processes  thereof,  from  ~59  cell  types  [54]),  it  is  perhaps  unrealistic  to  expect  that  on  the  order 
of  60-150  correlations  can  be  implemented  per  medullar  column.  It  is  therefore  pertinent  to  ask 
how  well  discriminations  can  be  made  based  on  smaller  subsets  of  these  correlations.  We  have 
performed  a  few  additional  simulations  and  optimizations  in  which  only  the  ‘most  informative’ 
correlations  were  retained.  These  were  judged  to  be  the  components  of  the  two  prototype  vectors 
(as  obtained  from  the  original  simulations)  with  the  largest  relative  differences  in  their 
magnitudes.  Figure  29  shows  the  results  of  these  experiments: 
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— ^  Motion-Motion 
-^Motion-Flicker 
-«^FIicker-HF  Flicker 


2  4  8  16  32  64 

Number  N  of  correlations  retained 

Figure  29:  Scoring  of  discrimination  as  a  function  of  the  number  of  correlations 
retained  in  the  receptive  field,  for  the  case  in  which  the  central  inter-unit 
discriminator  is  compared  with  each  of  the  left-side  discriminators.  (Note  the 
abscissa  is  on  a  log  scale.)  An  independent  optimization  was  performed  for  each 
datum. 

From  these  results,  we  conclude  that  even  on  the  order  of  ten  of  the  most  informative 
correlations  carry  substantial  information  for  discrimination  of  differing  spatiotemporal  statistics. 

What  is  the  nature  of  the  particular  products  that  by  our  criterion  are  the  most  informative? 
The  majority  of  them  involve  at  least  one  transient  signal,  and  interestingly,  all  of  them  involve 
at  least  one  delayed  signal.  In  addition,  there  is  substantial  overlap  between  the  sets  of  products 
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that  are  most  informative  for  each  of  the  three  different  input  scenarios  used  in  the  simulations. 
This,  like  the  results  in  Figure  29,  suggests  that  effective  and  general  discrimination  under  this 
scheme  could  be  accomplished  with  a  biologically  plausible  number  of  correlation  operations. 

Further  work  on  these  concepts,  including  consideration  of  moving  boundaries,  and  of  other 
input  scenarios  (including,  for  example,  those  used  in  our  experiments),  will  be  undertaken  in  the 
project  extension  if  granted. 

3.6.5  Winner-takes-all  neural  circuit  with  NMD  A  receptors 

In  simulation  and  analysis,  we  demonstrated  that  a  neuronal  network  with  lateral  feedback 
inhibition,  arranged  in  perhaps  the  simplest  possible  topology,  is  capable  of  realizing  the  winner- 
takes-all  (WTA)  function.  WTA  is  a  powerful  computational  primitive  that  is  often  attributed  to 
biological  nervous  systems.  The  neural  circuit  is  illustrated  in  Figure  30. 


Figure  30:  Members  of  a  set  of  competitive  neurons  Ni,  N2,  N3, ...  each  receives  a  respective 
excitatory  input  INi,  IN2,  IN3, ...,  via  an  NMDA  synapse  (closed  triangle).  These  neurons 
form  excitatory  synapses  (open  triangles)  onto  a  common  interneuron  Nfb?  which  projects 
feedback  to  all  neurons  in  the  set  via  inhibitory  synapses  (small  open  circles).  Although  not 
depicted,  the  outputs  of  neurons  Ni,  N2,  N3, ...  are  assumed  to  also  project  to  other  parts  of 
the  nervous  system. 

Excitatory  inputs  to  the  network  are  assumed  to  be  supplied  by  synapses  incorporating 
NMDA  receptors  (NMDARs),  a  class  of  glutamatergic  receptor  that  is  found  in  many  phyla  and 
frequently  associated  with  synapses.  Although  nonlinearities  such  as  neural  thresholding 
contribute  to  the  winner-takes-all  regimes  that  can  be  achieved  by  this  network,  these  depend 
most  critieally  on  the  unique  eleetrical  properties  of  the  NMDAR  ion  ehannels,  whieh  have  a 
nonmonotonic  current- voltage  relationship  under  physiological  conditions  [27],  with  a  negative 
slope  eonduetance  region  that  is  due  to  (kinetically  fast)  magnesium  bloekade.  This  eharacteristic 
renders  the  NMDAR  eapable  of  supporting  neural  amplifieation  and  bistability  in  conjunetion 
with  other  membrane  conduetanees  [49].  The  primary  finding  of  this  novel  work  is  that  a  WTA 
characteristic  can  be  induced  by  high-gain  regimes  that  result  from  such  interactions,  rather  than 
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requiring  a  high  intrinsic  or  parametric  gain  in  the  feedback  loops.  This  is  significant  because  it 
represents  a  mechanism  for  WTA  that  is  both  simple  and  at  the  same  time  biophysically 
plausible,  relying  on  known  characteristics  of  ubiquitous  classes  of  synaptic  receptors. 

A  simple  model  of  this  network  was  developed  for  numerical  analysis,  and  used  for  the 
simulations  reported  herein.  The  high-gain  behavior  that  is  associated  the  WTA  function  with  is 
enhanced  when  the  total  non-NMDAR  conductance  has  a  reversal  potential  below  typical  neural 
resting  potentials,  and  also  when  its  current-voltage  dependence  is  sub-linear  or  inward- 
rectifying  [47]  [49].  With  this  in  mind,  behavior  of  the  network  was  examined  in  simulations  for 
inhibitory  feedback  with  several  different  characteristics:  either  mildly  hyperpolarizing  (e.g.,  as 
might  be  expected  if  chloride  channels  were  associated  with  the  inhibitory  synapses)  or  more 
strongly  hyperpolarizing  (which  might  be  expected  with  potassium  channels),  and  having  either 
ohmic  or  inward-rectifying  channels.  Gamma-aminobutyric  acid  (GABA)-mediated  synapses, 
which  are  prime  candidates  to  implement  the  lateral  inhibition,  occur  in  two  major  classes  that 
reflect  different  combinations  of  these  characteristics  -  the  GABAb  class  involves  Kir  channels, 
which  conduct  potassium  and  are  inward-rectifying  (and  thus  might  be  expected  to  promote 
high-gain  behavior),  whereas  GABAa  receptor  channels  conduct  chloride  and  are  non-rectifying. 

Figure  31  below  illustrates  some  of  the  dc  transfer  characteristics  seen  in  this  network  for 
various  configurations: 
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Figure  31  (previous  page):  a  dc  input-output  characteristic  for  the  network  when  only  a 

single  competitive  neuron  Ni  is  activated.  The  inhibitory  feedback  synapses  are 

ohmic/strongly-hyperpolarizing.  The  abscissa  is  the  normalized  conductance  of  the 
input  (NMDA)  synapse  onto  the  neuron,  the  ordinate  is  its  membrane  potential,  and  the 
parameter  is  the  loop  gain  constant  which  is  varied  from  0  to  -5  in  decrements  of  -1. 

B-F:  dc  transition  characteristics  for  several  configurations  of  the  network  when  two 
neurons  receive  excitatory  input.  The  loop  gain  constant  —  —4  in  each  case.  Input  is 

swept  in  individual  trials,  and  the  parameter  is  the  input  to  the  second  neuron  N2,  which 

is  fixed  during  each  sweep  of  F^  but  varied  from  trial  to  trial.  Membrane  potential  of 

Ni  is  shown  in  black  and  ^2  in  g**ay.  In  B-D^  the  inhibitory  channels  are  ohmic  and 

F  is  stepped  from  0  to  35  in  increments  of  5.  In  B  the  inhibitory  reversal  potential  is 

-70mV;  in  C  and  D  it  is  -90mV.  C  shows  results  for  upward  sweeps  of,  while  D  shows 

for  sweeps  in  both  directions,  which  trigger  transitions  at  juxtaposed  limit  points  and 

illustrate  the  presence  of  hysteresis.  Directions  of  transitions  are  indicated  by  thin  arrows. 

The  heavy  arrow  at  lower  right  points  to  a  small  secondary  hysteresis  loop  (not  fully  visible 

at  this  resolution).  In  E  and  F  the  inhibitory  channels  are  inward-rectifying,  and  is  swept 
upward  in  both  cases.  In  E  the  reversal  potential  is  -70mV  and  stepped  from  0  to  35 

in  increments  of  5;  in  F  the  reversal  potential  is  -90mV  and  P2  is  stepped  from  0  to  17.5  in 
increments  of  2.5.  Resting  potential  in  all  cases  is  -60mV. 

Figure  31A  shows  the  de  input  (expressed  as  NMDAR  eonductance /"j  relative  to  the  resting 

membrane  eonduetance)  versus  output  (membrane  potential)  relationship  for  a  single  eompetitive 
neuron  Ni  when  that  neuron  is  the  only  one  in  the  network  receiving  excitatory  input.  The  loop 
gain  constant  is  the  parameter.  Membrane  potential  varies  smoothly  with  input  activation,  and  as 
might  be  expected,  the  slope  of  this  relationship  decreases  as  the  strength  of  inhibitory  feedback 
is  increased.  Figure  31  B-F  show  the  behavior  of  different  configurations  of  the  model  when  two 
of  the  competitive  neurons  receive  excitatory  input,  and  the  network  transitions  from  one 
‘winner’  to  another  as  the  first  input  exceeds  the  other.  (It  should  be  noted  that  this  transitional 
behavior  applies  not  just  to  the  two-input  case,  but  to  any  situation  in  which  the  two  neurons  with 
the  largest  inputs  are  the  only  ones  that  are  able  to  reach  a  state  of  depolarization).  When  the 
inhibition  is  ohmic  and  mildly  hyperpolarizing  (Figure  3 IB),  the  transitions  are  gentle  and  the 
network  cannot  reasonably  be  characterized  as  winner-takes-all.  In  the  remaining  cases, 
however,  there  are  input  ranges  for  which  the  transitions  are  not  only  sharp  but  discontinuous. 
These  discontinuities  and  the  hysteretic  effects  that  accompany  them  are  associated  with  bistable 
regimes.  Such  a  regime  prevails  over  the  range  of  P^  values  bounded  by  each  hysteresis  loop  in 

Figure  3  ID.  In  these  regions,  fixed-point  solutions  exist  in  which  either  neuron  may  be 
significantly  depolarized  (i.e.,  in  a  ‘high’  state),  while  the  other  is  either  hyperpolarized  or 
depolarized  to  a  lesser  extent  (in  a  ‘low’  state).  Which  of  the  two  solutions  might  be  assumed 
under  quasistatic  conditions  depends  on  the  history  of  excitation  of  the  system.  The 
discontinuous  jumps  depicted  in  Figure  31C-F  represent  limit  point  transitions,  which  occur  at 
fold  bifurcations  that  correspond  to  the  boundaries  of  bistable  regimes.  Outside  of  transitions 
between  winners,  the  state  of  the  winning  neuron  depends  on  its  input  level,  and  thus  carries 
analog  information  about  that  input. 
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The  fixed-point  solutions  in  this  network  model,  ineluding  its  bistable/WTA  regimes,  are 
shaped  by  a  rich  array  of  nonlinearities,  including  the  NMDAR  current-voltage  relationship  and 
the  neural  thresholding  function,  but  also  the  rectifying  characteristics  of  Kir  channels  (if 
present)  and  saturation  of  the  membrane  potential  in  a  ‘winning’  neuron.  Numerical  analysis  was 
used  to  characterize  these  in  more  detail,  and  shows  their  structure  can  be  quite  complex.  Some 
important  results  may  be  summarized: 

-  When  bistable  WTA  behavior  is  possible,  its  appearance  requires  some  minimum  value  of 
the  loop  gain; 

-  When  more  than  two  neurons  receive  excitatory  input,  then  mw/ti-stable  winner-takes-all 
states  are  possible; 

-  WTA  regimes  ultimately  collapse  and  vanish  when  the  inputs  become  sufficiently  large.  This 
is  due  to  the  fact  that  at  high  input  levels,  saturation  of  the  membrane  potential  in  the  winning 
neuron  begins  to  limit  the  recruitment  of  inhibition; 

-  When  WTA  behavior  is  supported,  behavior  is  qualitatively  similar  for  inward-rectifying  and 
non-rectifying  inhibition,  but  bi-/multi-stable  behavior  is  more  pronounced:  it  is  supported  at 
higher  inhibitory  reversal  potentials  and  at  lower  values  of  the  loop  gain  parameter  than  when 
the  inhibition  is  ohmic,  and  hysteresis  is  more  extensive  as  well. 

In  addition  to  the  dc  characteristic  of  the  model,  its  time-domain  and  ac  behavior  were  also 
examined.  An  important  question  is  whether  the  fixed  points  identified  by  the  stationary  analysis 
are  stable  or  unstable  in  nature;  the  answer  is  determined  by  the  dynamical  characteristics  of  the 
inhibitory  feedback  in  conjunction  with  the  input  state  and  other  network  parameters.  Analysis 
shows  that  the  network  may  be  prone  to  destabilization  when  there  are  two  or  more  independent 
states  in  the  feedback  loop  that  contribute  to  lags  in  its  response  -  and  this  is  certainly  the  case  in 
a  biologically-realistic  scenario  which  accounts  for  the  membrane  capacitance  of  the  inhibitory 
intemeuron  in  the  feedback  loop,  as  well  as  the  kinetics  of  synapses  in  the  feedback  path.  When 
these  lags  and  the  loop  gain  are  sufficiently  large,  destabilization  occurs  as  the  network  inputs 
approach  a  magnitude  that  supports  a  WTA  regime  -  but  interestingly,  the  network  tends  to  re¬ 
stabilize  at  some  point  as  inputs  increase  further.  This  is  illustrated  in  Figure  32: 
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Figure  32:  Graphs  illustrating  the  fixed-point  stability  characteristics  of  one  configuration  of 
the  WTA  network  when  a  single  input  is  active.  Inhibitory  synaptic  conductance  is  inward- 

rectifying  with  reversal  potential  -70mV.  The  resting  membrane  time  constant  in  the 

competitive  neuron  is  set  to  20ms;  membrane  charging  in  the  inhibitory  interneuron  is 
modeled  with  a  pole  with  time  constant  Sms,  and  is  cascaded  with  a  four-state  model  for 
GABAb  receptor  dynamics  [6]  that  introduces  three  additional  poles  in  the  feedback  path.  At 
left  are  depicted  the  real  parts  of  the  poles  of  the  closed-loop  system  at  its  fixed  points  as 

functions  of  the  input  strength  F  ,  in  this  case  for  =  — 4 .  Each  is  coded  by  a  different 

color.  Where  two  curves  overlie,  the  corresponding  poles  are  complex  conjugates.  The 
arrows  indicate  the  points  at  which  a  complex  pole-pair  transitions  between  left-  and  right- 
half-planes  and  bracket  an  input  range  over  which  the  fixed  points  are  unstable.  In  the 
middle  part  of  this  region,  the  poles  are  real  and  distinct,  implying  exponential  divergence  of 
trajectories  away  from  the  fixed  point  -  but  large-signal  limit  cycles  (i.e.,  oscillations)  are 
ultimately  assumed  by  such  trajectories.  At  right,  the  region  over  which  the  network  is 

unstable  in  the  space  spanned  by  F  and  is  shown  in  red  (for  purposes  of  comparison, 

the  pink  curve  shows  the  boundary  of  this  region  when  GABAa  kinetics  are  substituted  for 
GABAb).  In  the  blue  area  the  membrane  conductance  in  the  competitive  neuron  is  negative 
for  the  input  and  inhibitory  states  at  the  fixed  point,  and  in  the  green,  WTA  bistability 
would  be  possible  if  the  input  to  a  second  neuron  were  active  (and  feedback  instantaneous). 

The  results  in  Figure  32,  whieh  apply  when  a  single  neuron  is  aetive,  define  a  minimal  input 
range  over  whieh  the  network  is  unstable.  Further  analysis  shows  that  when  multiple  neurons  are 
aetivated,  instability  and  the  resulting  entrained  oseillations  may  oeeur  over  even  larger  input 
ranges.  Stability  is  lost  and  regained  in  all  eases  via  Hopf  bifurcations . 

This  seenario  of  destabilization  is  generally  ineompatible  with  WTA  funetionality  (at  least  as 
eoneeived  in  this  study)  when  it  applies.  It  is  most  pronouneed  (i.e.,  oeeurs  at  lower  loop  gains 
and  eovers  a  larger  range  of  input  strengths)  when  the  slow  dynamies  of  GABAb  reeeptors  (as 
opposed  to  faster  GABAa)  are  assoeiated  with  the  inhibitory  synapses  in  the  feedbaek  loops. 
Very  small  or  negative  membrane  eonduetanee  values  in  the  eompetitive  neurons,  whieh  are 
more  prevalent  when  the  inhibitory  eurrent  is  modeled  as  inward-reetifying,  also  promote 
instability.  Thus,  it  might  be  expeeted  that  lateral  inhibition  implemented  with  GABAb  synapses 
would  eompromise  the  ability  of  sueh  a  network  to  reliably  implement  the  WTA  funetion. 
However,  an  important  additional  finding  is  that  other  biologieally-plausible  eharaeteristies  ean 
mitigate  against  unstable  behavior.  When  a  simplifying  assumption  of  linearity  in  the  inhibitory 
intemeuron  (used  in  most  simulations)  is  relaxed  and  saturating  effeets  introdueed,  it  ean  squeleh 
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large-signal  oscillations  and  establish  a  reliable  WTA  regime  (although  the  network  may  pass 
through  an  unconditionally  unstable  regime  at  lower  input  values  to  reach  it).  In  addition,  the 
presence  of  GABAa  as  well  as  GABAb  receptors  in  the  inhibitory  feedback  pathway  -  a 
possibility  supported  by  evidence  from  the  biological  literature  [1]  -  can  have  a  significant 
stabilizing  effect. 

This  is  a  brief  summary  of  results  reported  in  greater  detail  in  a  paper  appearing  in  the 
']oumdX  Frontiers  in  Computational  Neuroscience. 

4  Personnel  Supported 

In  Karin  Nordstrom’s  lab,  AFOSR  funds  supported  Ph.D.  students  Yu-Jen  (Frank)  Lee  and 
Olga  Dyakova.  Lee  defended  his  thesis  in  May  2014,  with  Prof  Holger  Krapp  as  the  Faculty 
opponent.  Lee  and  Nordstrom  performed  most  of  electrophysiology  presented  in  this  report, 
complemented  by  behavioral  and  analytical  work  performed  by  Dyakova,  who  enrolled  as  a 
Ph.D.  student  in  September  2013. 

In  Mark  Frye’s  lab,  AFOSR  funds  supported  Ph.D.  students  Jacob  Aptekar  and  Mehmet 
Keles,  and  M.S.  student  Ross  Kelley.  Aptekar,  who  graduated  in  June  2014,  performed  flight 
behavioral  analyses,  neurogenetics,  and  two-photon  imaging.  Kelley,  who,  graduated  in  June 
2013,  performed  flight  behavioral  analyses  and  neurogenetics.  The  funds  currently  support 
Keles,  who  is  performing  flight  behavioral  analyses,  two-photon  imaging,  and  neurogenetics. 

At  Tanner  Research,  AFOSR  funds  partially  supported  Patrick  Shoemaker,  senior  scientist, 
who  has  worked  on  analytical  techniques,  theory,  and  modeling,  as  well  as  on  supporting 
experimental  design. 

5  Publications 

Seventeen  papers  for  publication  in  the  peer-reviewed  literature  have  appeared,  or  are  under 
production,  with  the  full  or  partial  support  of  this  grant. 

5.1  Peer-Reviewed  Publications  Fully  or  Partially  Supported  by  this  Grant: 

Aptekar  JW,  Keles  MF,  Mongeau  J-M,  Lu  PM,  Frye  MA,  Shoemaker  PA  (2014)  Method  and  software 
for  using  m-sequenees  to  eharaeterize  parallel  eomponents  of  higher-order  visual  traeking  behavior 
'm  Drosophila.  Front  Neurosci  vol.  8,  DOI:  10.3389/fncir.2014.00130. 

Aptekar  JW,  Frye  MA  (2013)  Higher-order  figure  discrimination  in  fly  and  human  vision,  Curr  Biol 
23(16):R694-700. 

Aptekar  JW,  Shoemaker  PA,  Frye  MA  (2012)  Figure  tracking  by  flies  is  supported  by  parallel  visual 
streams.  Curr  Biol  22:482-487. 

De  Haan  R,  Lee  Y-J,  Nordstrom  K  (2013)  Novel  flicker-sensitive  visual  circuit  neurons  inhibited  by 
stationary  patterns,  JA/ewrosci,  33  (21):  8980-8989. 

Duisteimars  BJ,  Care  RA,  Frye  MA  (2012)  Binocular  interactions  underlying  the  classic  optomotor 
responses  of  flying  flies.  Front  Behav  Neurosci  6(6),  DOI:  10.3389/fnbeh 

Fox  JL,  Aptekar  JW,  Zolotova  NM,  Shoemaker  PA,  Frye  MA  (2014)  Figure-ground  discrimination 
behavior  in  Drosophila.  I.  Spatial  organization  of  wing-steering  responses.  J  Exp  Biol  217:  558- 
569. 

Fox  JL,  Frye  MA  (2014)  Figure-ground  discrimination  behavior  in  Drosophila.  II.  Visual  influences  on 
head  movement.  J  Exp  Biol  217:  570-579 
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Frye  MA  (2013)  Visual  attention:  A  cell  that  focuses  on  one  object  at  a  time.  Curr  Biol  23:  R61-63 

Lee  Y-J  and  Nordstrom  K  (2012)  Higher  order  motion  sensitivity  in  fly  visual  circuits.  PNAS,  109  (22): 
8758-8763. 

Shoemaker  PA  (2015)  Neuronal  networks  with  NMDARs  and  lateral  inhibition  implement  winner-takes- 
all.  Front  Neurosci  9(12),  DOI:  10.3389/fncom.2015.00012 

Wasserman  S,  Lu  P,  Aptekar  JW,  Frye  MA  (2012)  Flies  dynamically  anti-track,  rather  than  ballistically 
escape,  aversive  odor  during  flight.  J Exp  Biol  215:2833-2840. 

Wasserman  SM,  Aptekar  JW,  Lu  P,  Nguyen  J,  Wang  AL,  Keles  MF,  Grygoruk  A,  Krantz  DE,  Larsen  C, 
and  Frye  MA  (2015)  Olfactory  neuromodulation  of  motion  vision  circuitry  in  Drosophila.  Curr 
Biol  yol  24,  pp.  467-472. 

Papers  Submitted  or  in  Production,  Fully  or  Partially  Supported  by  this  Grant: 

Aptekar  JW,  Zolotova  NM,  Frye  MA  (in  production)  The  switch  from  object  tracking  to  aversion  is 
driven  by  higher-order  feature  detection  in  Drosophila. 

Aptekar  JW,  Keles  MF,  Lu  PM,  Zolotova  NM,  Frye  MA  (in  production)  Optic  glomeruli  encode  higher- 
order  visual  features  in  Drosophila.  To  be  submitted.  Nature  Neuroscience. 

Dyakova  O,  Lee  Y-J,  Nordstrom  K  Natural  scene  statistics  directly  affect  the  response  properties  of 
higher-order  visual  circuit  neurons.  In  resubmission,  Nature  communications. 

Lee  Y-J,  Nordstrom  K  (in  production)  Figure  motion  suppresses  the  response  to  wide-fleld  optic  flow  in 
hoverfly  tangential  cells. 

Lee  Y-J,  Jonsson  O  and  Nordstrom  K  (2015)  Spatio-temporal  dynamics  of  impulse  responses  to  figure 

motion  in  optic  flow  neurons.  Resubmitted  to  PLoS  ONE. 

5.2  Papers  by  the  Principals  in  Related  Areas: 

Fox  JL,  Frye  MA  (2013)  Animal  behavior:  Fly  flight  moves  forward.  Curr  Biol  vol.  23(7):  R278-279 

Hidayat  E,  Medvedev  A  and  Nordstrom  K  (2013)  “On  identification  of  elementary  motion  detectors”. 
Proceedings  of  the  International  Symposium  on  Computational  Models  for  Life  Sciences 
(CMLS’2013). 

Hidayat  E,  Medvedev  A  and  Nordstrom  K  (2013)  Laguerre  domain  identification  of  the  elementary 
motion  detector  model  in  insect  vision.  Proceedings  of  the  IF  AC  International  Workshop  on 

Adaptation  and  Learning  in  Control  and  Signal  Processing  (ALCOSP’2013).  1 1  (1):  623-628.  DOI 
10.31 82/20 1 3 0703  -3  -FR-403  8 .00070. 

Nordstrom  K  and  Gonzalez-Bellido  PT  (2013)  “Invertebrate  vision:  Peripheral  adaptation  to  repeated 
object  motion”.  Curr  Biol  yol  23  (15):  R655-R656.  DOI  10.1016/j.cub.2013.06.009. 

Nordstrom  K  (2013)  “Robust  prey  detection  in  a  small  nervous  system”.  PNAS  vol.  110  (2):  389-390. 
DOI  www.pnas.org/cgi/doi/10.1073/pnas.1219973110. 

Dunbier  JR,  Wiederman  SD,  Shoemaker  PA,  O’Carroll  DC  (2012)  Facilitation  of  dragonfly  target 
detectors  by  slow  moving  features  on  continuous  paths.  Frontiers  in  Neural  Circuits,  vol.  6,  doi: 
10.3389/fncir.2012.00079. 

O’ Carroll  DC,  Barnett  PD,  and  Nordstrom  K  (2012)  “Temporal  and  spatial  adaptation  of  transient 
responses  to  local  features”.  Front  Neural  Circuits,  6:  74.  DOI  10. 3389/fncir.2012. 00074.  IF:  5.3. 
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Shoemaker  PA  (2014)  Implementation  of  visual  motion  detection  in  analog  ‘neuromorphic’  circuitry  -  a 
case  study  of  the  issue  of  circuit  precision.  Invited  paper,  Proceedings  of  the  IEEE,  vol.  102,  pp. 
1557-1570. 

Shoemaker  PA,  Wiederman  SD,  and  O’Carroll,  DC  (2013)  Can  a  Competitive  Neural  Network  Explain 
Selective  Attention  in  Insect  Target  Tracking  Neurons?  Proceedings,  6th  International  IEEE  EMBS 
Conference  on  Neural  Engineering,  (San  Diego,  Nov.  2013) 

Wasserman  S,  Lu  P,  Aptekar  JW,  Frye  MA  (2012)  Flies  dynamically  anti-track,  rather  than  ballistically 
escape,  aversive  odor  during  flight.  J Exp  Biol  vol.  215:2833-2840 

Wiederman  SD,  Shoemaker  PA,  and  O’Carroll  DC  (2013)  Correlation  between  OFF  and  ON  channels 
underlies  dark  target  selectivity  in  an  insect  visual  system.  Journal  of  Neuroscience  vol.  33:  13225- 
13232. 

6  Interactions  /  Transitions 

6.1  Participation/presentation  at  meetings,  conferences,  and  seminars 

6.1.1  Service  on  Organizing  Committees  for  Meetings: 

Nordstrom  organized  a  symposium  entitled  “Activity  generated  modulation  of  motion  vision  responses” 
at  the  10^^  International  Congress  of  Neuroethology  in  College  Park,  MD,  August  2012. 

Nordstrom  co-organized  the  Insect  Vision:  Cells,  Computation,  and  Behavior  meeting,  at  HHMI  Janelia 
Farm  Research  Campus,  March  2013. 

Nordstrom  was  a  scientific  advisor  for  the  International  Conference  on  Invertebrate  Vision  in 
Backaskog,  Sweden,  August  2013. 

Nordstrom  is  presently  co-organizing  the  next  Insect  Vision:  Cells,  Computation,  and  Behavior  meeting, 
scheduled  to  take  place  at  HHMI  Janelia  Farm  Research  Campus,  April  2015. 

6.1.2  Conference  Presentations  /  Posters: 

A  poster  by  Fox  and  Frye  entitled  Figure-ground  discrimination  in  flying  Drosophila  was  presented  at 
the  Neurobiology  of  Drosophila  meeting.  Cold  Spring  Harbor  Labs,  October  2011. 

Nordstrom  gave  an  invited  research  seminar  called  Salient  feature  tracking  at  the  Max  Planck  Institute 
for  Chemical  Ecology,  Jena,  Germany,  2012. 

Shoemaker,  Nordstrom,  and  Fox  attended  the  10^^  International  Congress  of  Neuroethology  in  College 
Park,  MD  in  August  2012.  Fox  gave  a  talk  entitled  Two  distinct  visual  microcircuits  required  for  figure- 
ground  discrimination  in  flies.  Nordstrom  presented  a  poster  entitled  Bar  cells:  Underlying  neuro¬ 
physiological  mechanisms. 

A  poster  by  Aptekar  and  Frye  entitled  The  horizontal  system  cells  of  the  lobula  plate  in  Drosophila  have 
dendritic  sub-regions  that  respond  to  figure  motion  in  both  the  preferred  and  anti-preferred  directions 
against  a  moving  ground  was  presented  at  the  Society  for  Neuroscience  meeting.  New  Orleans,  LA, 
November  2012. 

A  poster  by  Kelley,  Fox,  Aptekar,  and  Frye  entitled  Two  distinct  visual  microcircuits  required  for  figure- 
ground  discrimination  in  flies  was  presented  at  the  Society  for  Neuroscience  meeting.  New  Orleans,  LA, 
November  2012. 

A  poster  by  Aptekar,  Fox,  and  Frye  entitled  The  horizontal  system  of  cells  in  the  lobula  plate  of 
Drosophila  have  dendritic  subregions  that  respond  to  figure  motion  in  both  preferred  and  anti-preferred 
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directions  against  a  moving  ground  was  presented  at  the  Society  for  Neuroscience  annual  meeting,  New 
Orleans,  LA,  November  2012. 

Nordstrom,  Frye  and  Fox  presented  ongoing  work  at  the  Insect  Vision:  Cells,  Computation,  and 
Behavior  meeting,  HHMI  Janelia  Farm  Research  Campus,  March  2013.  Nordstrom’s  talk  was  entitled 
Feature  detection  by  hoverfly  LPTCs. 

Nordstrom  gave  a  talk  entitled  Neural  mechanisms  underlying  target  tracking  at  the  Dynamics  of  Prey 
Capture  and  Escape  Meeting,  at  HHMI  Janelia  Farm  Research  Campus,  March  2013. 

Frye  gave  a  plenary  talk  entitled  Olfactory  feature  detection  in  flies  at  the  Tata  Institute  of  Fundamental 
Research,  Genes,  Circuits  and  Behavior,  Mumbai,  India,  March  2013. 

Frye  gave  an  invited  research  seminar  QntitlQd  Algorithms  and  neurons  for  figure-ground  discrimination 
in  Drosophila  MRC  Centre  for  Developmental  Biology,  King’s  College  London,  UK,  April  2013. 

Nordstrom  gave  a  presentation  Figure-ground  discrimination  in  hoverfly  tangential  neurons  as  an 
invited  lead  speaker  at  the  Rank  Prize  Funds:  Symposium  on  the  Computational  Basis  of  Early  Vision. 
The  Wordsworth  Hotel,  Grasmere,  Cumbria,  UK,  May  2013. 

Frye  gave  an  invited  seminar  Olfactory  feature  detection  in  flies  at  the  Department  of  Biology,  University 
of  Indiana,  Bloomington,  May  2013 

Frye  gave  the  presentation  Multi-sensory  feature  detection  at  the  Center  for  Neural  and  Emergent 
Systems,  HRL  Laboratories  LLC,  Malibu,  CA,  May  2013. 

Frye  gave  the  presentation  Higher-order  visual  feature  detection  at  the  Cell  Press  Symposium:  Genes, 
Circuits  and  Behavior,  Toronto  CN,  June  2013. 

Frye  gave  a  plenary  lecture  Visual  course  control  and  feature  detection  during  flight  in  Drosophila  and 
Shoemaker  presented  a  poster  entitled  A  biophysical  model  for  neural  'delay-and-corr elate’  at  the 
International  Conference  on  Invertebrate  Vision  in  Backaskog,  Sweden,  August  2013.  Both  Pis  interacted 
with  AFOSR  Emeritus  Willard  Larkin. 

Nordstrom  gave  the  presentation  Neural  mechanisms  underlying  hoverfly  target  detection,  and  Lee 
presented  Edge  dependence  of  inhibition  by  stationary  patterns  in  cSIFE  in  the  hoverfly  lobula  plate  at 
Visionarium  XII,  Tvarminne  Zoological  Station,  Finland,  October  2013. 

A  poster  by  Yen  and  Nordstrom  entitled  A  Novel  Stationary  Pattern  Inhibited  High-Speed  Motion 
Sensitive  Visual  Neuron  in  the  Lobula  Plate  was  presented  at  the  Society  for  Neuroscience  meeting  in  San 
Diego,  December  2013. 

Nordstrom  gave  an  invited  research  seminar  Processing  of  naturalistic  features  in  the  hoverfly  visual 
system  at  the  Max  Planck  Institute  for  Neurobiology,  Martinsried,  Germany,  2014. 

Frye  gave  a  plenary  talk  Systems  identification  of  visual  figure  detection  in  flies  at  the  Computational 
Neuroscience  meeting  (CNS):  Quebec  City,  CN,  June  2014. 

Dyakova  gave  a  poster  presentation  entitled  Second-order  statistics  of  natural  scenes  influence  object 
detection  in  hoverfiies)  to  the  Lund  Vision  Group,  Lund  University,  August  2014. 

Frye  gave  the  Aubrey  Gorbman  Lecture,  Olfactory  neuromodulation  of  motion  vision  behavior  and 
circuitry  in  Drosophila,  Department  of  Biology  at  University  of  Washington,  October  2014. 

6.2  Consultative  and  advisory  functions 

Nordstrom  aeted  as  an  aeademie  advisor  for  Lee  and  Dyakova,  and  Frye  for  Aptekar,  Kelley,  and 
Keles,  under  support  of  the  grant.  Shoemaker  eonsulted  for  the  Frye  laboratory  via  the  Howard 
Hughes  Medieal  Institute. 
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6.3  Transitions 

Yu-Jen  (Frank)  Lee  graduated  from  Uppsala  University  with  a  Ph.D. 

Jacob  Aptekar  graduated  from  UCLA  with  a  Ph.D.,  and  is  now  a  Postdoctoral  Research 
Associate  at  UCLA. 

Ross  Kelley  graduated  from  UCLA  with  a  M.S.,  and  is  now  a  medical  student  at  the  University 
of  Arizona. 

Jessica  Fox  completed  her  postdoctoral  fellowship  at  UCLA,  and  is  now  Assistant  Professor  and 
has  started  her  own  laboratory  at  Case  Western  Reserve  University. 
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